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This report reviews the characteristics 
of 21 instruments that measure stu- 
dent engagement in upper elementary 
through high school. It summarizes what 
each instrument measures, describes its 
purposes and uses, and provides tech- 
nical information on its psychometric 
properties. 

Researchers, educators, and policymakers 
are increasingly focused on student engage- 
ment as the key to addressing problems 
of low achievement, student boredom and 
alienation, and high dropout rates (Fred- 
ricks, Blumenfeld, and Paris 2004). To 
increase student engagement, educators and 
evaluators need to understand how engage- 
ment has been defined and to assess the op- 
tions for measuring it. However, instruments 
for measuring engagement are not easily 
accessible as a group in a way that allows for 
comparison because they arise from differ- 
ent disciplinary perspectives and theoretical 
frameworks. 

This report summarizes the characteristics of 
instruments that measure student engagement 
in upper elementary through high school, 
providing information on the range of instru- 
ments available. It is not a technical review of 
the quality of these measures. 



The findings are organized in response to two 
questions addressed by the study: 

• What instruments are available to mea- 
sure student engagement in upper elemen- 
tary through high school? 

• What are the characteristics of each iden- 
tified measure? 

The report describes the results of a litera- 
ture review to identify available instruments. 
The 21 instruments identified are described 
according to what is measured, their pur- 
pose and use, and the technical information 
available on their psychometric properties. 

The instruments include 14 student self-report 
instruments, 3 teacher reports on students, 
and 4 observational measures 

• What is measured. The constructs assessed 
can be described by the extent to which 
the instruments represent the multidi- 
mensional nature of engagement (behav- 
ioral, emotional, and cognitive) and the 
object of engagement. Of the 14 student 
self-report instruments described, 5 as- 
sess all three dimensions of engagement, 

5 assess two dimensions, and 4 assess one 
dimension. Nine are worded to reflect 
general engagement in school, and five are 
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worded for use at the class level. Two of 
the three teacher report instruments can be 
used by teachers for reporting on student 
engagement in any subject and the third 
for reporting on engagement in reading. 
Two of the four observation measures 
provide a coding system for observing 
an individual student’s on- and off-task 
behavior or engaged time in classroom 
settings, and two assess classroom engage- 
ment across all students in the class. 

• Purpose and use. The 21 instruments 
have several different purposes and uses, 
including research on motivational and 
cognitive theories of learning; research 
on disengagement and dropping out; 
evaluation of school reform efforts and 
interventions; monitoring of engagement 
at the teacher, school, or district level; 
diagnosis and monitoring at the student 
level; and needs assessment of students’ 



developmental assets (the relationships, 
opportunities, and personal qualities 
that young people need to avoid risks and 
enhance positive outcomes). 

• Technical information on psychomet- 
ric properties. Reliability and validity 
information was found for all but one 
instrument. Overall, developers reported 
internal consistency results for student 
self-report and teacher report measures 
that were at or near acceptable levels for 
use, ranging from .49 to .93, with most 
scales at .70 to .80. Substantial information 
was also available on validity. For exam- 
ple, 13 measures had positive correlations 
with measures of student achievement. 
This report does not judge whether the 
technical information accessed is sufficient 
for any particular use of an instrument. 

January 2011 
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This report reviews 
the characteristics 
of 21 instruments 
that measure 
student 
engagement in 
upper elementary 
through high 
school. It 
summarizes what 
each instrument 
measures, 
describes 
its purposes 
and uses, and 
provides technical 
information on 
its psychometric 
properties. 



WHY THIS STUDY? 

Researchers, educators, and policymakers are 
focusing more on student engagement as the key 
to addressing low achievement, student boredom 
and alienation, and high dropout rates (Fredricks, 



Blumenfeld, and Paris 2004). As schools and dis- 
tricts seek to increase engagement, it is important 
for them to understand how it has been defined 
and to assess the options for measuring it. 

One challenge educators and evaluators face in 
measuring engagement is determining the appro- 
priateness of the available instruments, especially 
given limited time to review the literature. Instru- 
ments for measuring engagement also reflect 
different disciplinary perspectives and theoretical 
frameworks and are thus not easily compared. 

To address the information needs of education 
professionals, this report describes the 21 instru- 
ments for measuring engagement in upper el- 
ementary through high school identified through 
a literature review. The report does not include a 
technical review of the quality of each measure, 
nor does it recommend or identify strengths or 
weaknesses of particular instruments. 



What is student engagement? 

Interest in student engagement has grown over 
the past two decades, although there is substantial 
variation in how it has been defined and mea- 
sured. Early studies defined student engagement 
primarily by observable behaviors such as partici- 
pation and time on task (Brophy 1983; Natriello 
1984). Researchers have also incorporated emo- 
tional or affective aspects into their conceptual- 
ization of engagement (Connell 1990; Finn 1989). 
These definitions include feelings of belonging, 
enjoyment, and attachment. More recently, re- 
searchers have studied aspects of cognitive engage- 
ment, such as students’ investment in learning, 
perseverance in the face of challenges, and use 
of deep rather than superficial strategies (Fred- 
ricks, Blumenfeld, and Paris 2004). Some have 
also included self-regulation (the extent to which 
students demonstrate control over their learning 
actions) as a component of cognitive engagement 
(Pintrich and DeGroot 1990; Miller et al. 1996). 

Researchers have proposed theoretical models 
suggesting that student engagement predicts 
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Inclusion of engagement 
as a goal of school 
improvement, growing 
awareness of the 
connection between 
disengagement and 
dropping out, and use 
of engagement as a 
program or intervention 
outcome all help explain 
the increased interest 
in understanding 
and collecting data 
on engagement 



subsequent achievement and suc- 
cess in school. One of the earliest 
theories of engagement was the 
participation-identification model 
(Finn 1989). This theory defines 
engagement in school as “having 
both a behavioral component, 
termed participation , and an emo- 
tional component, termed iden- 
tification [emphasis in original]” 
(Finn and Voelkl 1993, p. 249). 



Another influential model was 
developed by Connell and his 
colleagues (Connell 1990; Con- 
nell and Wellborn 1991; Skinner 
and Belmont 1993), who distinguish two ends of 
a continuum: engagement and disaffected pat- 
terns of action. Engaged students show behavioral 
involvement in learning and positive emotional 
tone; they persevere in the face of challenge (Con- 
nell 1990; Connell and Wellborn 1991). In contrast, 
disengaged or disaffected students are passive, do 
not try hard, are bored, give up easily, and display 
negative emotions, such as anger, blame, and 
denial (Skinner and Belmont 1993). 



• Cognitive engagement is defined as the 
student’s level of investment in learning; it 
includes being thoughtful and purposeful in 
the approach to school tasks and being willing 
to exert the effort necessary to comprehend 
complex ideas or master difficult skills (Fred- 
ricks, Blumenfeld, and Paris 2004). 



Why interest in engagement has increased 

Several factors may explain the increased interest 
in understanding and collecting data on engage- 
ment. Among these are the inclusion of engage- 
ment as a goal of school improvement, growing 
awareness of the connection between disengage- 
ment and dropping out, and use of engagement as 
a program or intervention outcome. 

Engagement as a goal of school improvement. 

Student engagement measures have been shown to 
correlate positively with achievement and nega- 
tively with the likelihood of dropping out of school 
(Fredricks, Blumenfeld, and Paris 2004). Engaged 
students are more likely to earn better grades and 
perform well on standardized tests (Fredricks, 
Blumenfeld, and Paris 2004; Marks 2000). 



In a review of the literature on student engage- 
ment, Fredricks, Blumenfeld, and Paris (2004) 
propose that student engagement has multiple 
dimensions: behavioral, emotional, and cognitive. 

• Behavioral engagement draws on the idea of 
participation and includes involvement in 
academic, social, or extracurricular activi- 
ties; it is considered crucial for achieving 
positive academic outcomes and preventing 
dropping out (Connell and Wellborn 1990; 
Finn 1989). 

• Emotional engagement focuses on the extent 
of positive (and negative) reactions to teach- 
ers, classmates, academics, and school. 
Positive emotional engagement is presumed 
to create student ties to the institution and in- 
fluence students’ willingness to work (Connell 
and Wellborn 1990; Finn 1989). 



Engagement has been shown to decline as students 
progress through the upper elementary grades and 
middle school, reaching its lowest levels in high 
school (Marks 2000; National Research Council 
and Institute of Medicine 2004). This decline can 
be even more dramatic as students move through 
feeder patterns of low-performing, high-poverty 
schools (Yazzie-Mintz 2007). Some studies 
estimate that by high school as many as 40-60 
percent of youth are disengaged (Marks 2000). Not 
surprisingly, increasing student engagement has 
been an explicit goal of many school and district 
improvement efforts, especially at the secondary 
level (National Research Council and Institute of 
Medicine 2004). Measurement is required if prog- 
ress is to be tracked over time. 

Connection between disengagement and drop- 
ping out. Measuring engagement helps identify 
at-risk students. For many students, dropping out 
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of high school is the last step in a long process of 
disengagement (Finn 1989). Its consequences for 
middle and high school students from disadvan- 
taged backgrounds are especially severe, because 
these students are less likely to graduate and will 
face more limited employment prospects, increas- 
ing their risk of poverty, poor health, and involve- 
ment in the criminal justice system (National 
Research Council and Institute of Medicine 2004). 
For this reason, many educators, school psycholo- 
gists, and community organizations are inter- 
ested in obtaining better data on engagement and 
disengagement for needs assessment, diagnosis, 
and prevention. 

Engagement as a program or intervention outcome. 

As part of the increased focus on school account- 
ability over the past 15 years, more attention has 
been paid to studying and reporting the effective- 
ness of interventions designed to improve student 
outcomes. Currently, many school reform models, 
programs, and student interventions focus on en- 
hancing engagement to improve achievement and 
school completion rates. Examples of interventions 
that have identified and measured engagement as 
an important student outcome include: 

• The Institute for Research and Reform in 
Education (IRRE) has worked in nine districts 
nationwide to implement First Things First, a 
school reform model in which schools commit 
to improving engagement and strengthening 
relationships between students and adults 
(http://www.irre.org). IRRE assists schools 

in collecting meaningful data on student 
engagement. 

• Check and Connect is aimed at students iden- 
tified as at risk of dropping out (http://www. 
ici.umn.edu/checkandconnect). The program 
is designed to improve engagement by maxi- 
mizing personal contact and opportunities to 
build trusting relationships with a mentor or 
monitor. Behavioral engagement (as reflected 
in attendance, grades, and suspensions) is 
checked regularly and used to help mentors 
strengthen students’ connection with school. 



The National 
Center for School 
Engagement (NCSE) 
partners with 
school districts, law 
enforcement agen- 
cies, courts, and state 
and federal agencies 
to support youths 
and their families in 



To increase engagement 
and find solutions for low 
academic achievement 
and high dropout rates, 
education professionals 
need to understand how 
engagement has been 
defined and to assess the 
options for measuring it 



improving engage- 
ment (http://www.schoolengagement. org). 
NCSE supports truancy reduction programs 
and helps schools track data on attendance 
and school engagement. 



What this study examines 

In seeking to increase engagement and find solu- 
tions for low academic achievement and high 
dropout rates, education professionals need to 
understand how engagement has been defined and 
to assess the options for measuring it. This report 
summarizes the characteristics of instruments 
used to measure student engagement in upper 
elementary through high school (see appendix A 
for instrument abstracts). 

Using a methodology described briefly in box 1 
and at length in appendix B, this study addresses 
two primary research questions: 

1. What instruments are available to measure 
student engagement in upper elementary 
through high school? 

2. What are the characteristics of each identified 
measure? 

The report describes 21 instruments available for 
use at the upper elementary through secondary 
level (box 2 defines the three types of instru- 
ments). It focuses on this age range because 
of the documented decline in motivation and 
engagement across middle and high school (Na- 
tional Research Council and Institute of Medi- 
cine 2004). 




4 



MEASURING STUDENT ENGAGEMENT IN UPPER ELEMENTARY THROUGH HIGH SCHOOL: 21 INSTRUMENTS 



BOX 1 

Methodology 

The instruments included in this re- 
port were identified through rigorous 
processes of searching and screening 
the literature and other sources for 
instrument names and summarizing 
information on the identified instru- 
ments (see appendix A for instrument 
abstracts and appendix B for more 
detail on the methodology). 

Searching and screening. Databases 
including Academic Search Premier, 
PsycINFO, and ERIC were searched 
for student engagement instruments 
using systematic keyword searches 
(see table B1 in appendix B). The goal 
was to find articles that use the word 
engagement in their description of 
what was measured. Although there 
is some overlap in the meaning of 
engagement and other closely related 
terms such as school belongings bond- 
ings and student motivation , the search 
was limited to the term engagement , 
because it often has a particular mean- 
ing to practitioners as an important 
school goal or intervention outcome. It 
should be noted that several reviews of 
the engagement literature (Jimerson, 
Campos, and Grief 2003; Fredricks 
et al. 2004) have pointed out the lack 
of clear and accepted definitions of 
and distinctions between engagement 
and other related terms. 

The search, restricted to studies 
published between 1979 (to predate 



the earliest emergence of engagement 
studies in the early 1980s) and May 
2009, resulted in 1,314 citations. All 
were reviewed to exclude off-topic 
citations and identify any named 
instruments. Citations coded as on 
topic yielded 144 named instruments; 
12 more were identified through 
supplementary processes, for a total 
of 156. Seven criteria were used in 
excluding instruments, resulting in 
137 excluded instruments (see figure 
B1 in appendix B): 

• Forty-six were intended for 
student populations outside the 
study age range. 

• Five were used only with special 
education populations. 

• Twelve were developed and used 
before 1979. 

• Thirty-one measured a construct 
other than engagement (see table 
B4). 

• Eleven were large-scale surveys 
that included only a few items 
on student engagement (see table 
B5). 

• Twenty had limited or confusing 
information that made com- 
pleting an accurate description 
difficult (see table B6). 

• Twelve were excluded for other 
reasons. 



Following these exclusions, 19 of the 
original 156 named instrument or in- 
strument packages were determined to 
be appropriate for inclusion. Two con- 
tained separate measures of engage- 
ment— one student self-report measure 
and one teacher report measure. The 
two separate measures are described in 
one instrument abstract in appendix A 
because they have the same instrument 
name. Thus, there are 19 instrument 
abstracts, but 21 instruments are 
described in the findings. 

Finding and summarizing informa- 
tion. In addition to citations for each 
instrument located through the 
initial search, further searches were 
conducted on each instrument name, 
abbreviation, and author to uncover 
any additional materials. Information 
was then systematically summarized 
using an instrument-documentation 
protocol (see table B7 in appendix B), 
and a draff abstract was prepared for 
each instrument detailing availability, 
population, method type, background, 
administration, what is measured, 
scoring/reporting, reliability, valid- 
ity, and use. The instrument abstracts 
underwent three levels of review 
to ensure accuracy. The completed 
abstracts were sent to the instrument 
developers/authors to review for accu- 
racy. Developers provided feedback on 
18 of the 19 abstracts, offering minor 
changes to the descriptions or updated 
information and additional refer- 
ences or otherwise indicating that the 
abstract information was accurate. 
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BOX 2 

Definitions of key terms 

Three types of measurement methods 
are discussed in this report: 

Student self-reports are measures in 
which students respond to items using 
specified response formats (such as 
“strongly agree” to “strongly disagree” 
or “very true of me” to “not true of 
me”). Scores can be summed or aver- 
aged across items to form subscale or 
total scores to describe the student. 

Teacher reports are scores assigned to 
students based on teacher responses 
to a set of items using a specified 
response format (for example, “very 
true of student” to “not true of 
student”). 

Observational measures involve 
direct observation of behavior of 



individuals, targeted students, or 
classrooms. This study includes only 
systematic observational measures 
that use predetermined coding 
systems to record observations. Ob- 
servational methods require trained 
observers to collect the data accu- 
rately and as intended by the instru- 
ment developer. 

A construct is a variable that cannot 
be observed directly but is as- 
sumed to exist on the basis of other 
evidence. (The term is not normally 
applied to directly observable and 
measurable behaviors, such as at- 
tendance or suspension rates.) For 
example, the variable “emotional 
engagement” cannot be directly seen, 
but it is hypothesized to exist and 
to influence other behaviors. When 
an instrument is used to measure a 
construct, evidence must be obtained 
to show that the instrument actually 



measures the abstract or hypothetical 
construct it is intended to measure. 
Construct validity refers to the degree 
to which an instrument actually 
measures a construct. 

A scale is a set of items or ques- 
tions intended to measure the same 
construct. A scale score is created 
by summing or averaging the scores 
on the individual items. Some 
instruments, including many of the 
student self-report instruments, 
measure multiple constructs and 
thus have multiple scales (in which 
case they may be called subscales). 
For example, the Research Assess- 
ment Package for Schools contains a 
student self-report engagement scale, 
which has two subscales, Ongoing 
Engagement in School and Reaction 
to Challenge. Items on each subscale 
can be summed to create subscale 
scores. 



WHAT INSTRUMENTS ARE AVAILABLE FOR 
MEASURING STUDENT ENGAGEMENT IN UPPER 
ELEMENTARY THROUGH HIGH SCHOOL? 

The 21 measures of student engagement (14 student 
self-report instruments, 3 teacher report instru- 
ments, and 4 observation instruments) are listed in 
table 1. Three measures (Classroom AIMS, the Code 
for Instructional Structure and Student Academic 
Response [MS-CISSAR], and Engagement versus 
Disaffection with Learning [EvsD]) were developed 
for use with elementary school populations but 
have also been used with middle and high schools 
students. Two instruments (the Student Engage- 
ment Measure [SEM]-MacArthur and the Read- 
ing Engagement Index [REI]) were developed for 
use with upper elementary students and teachers; 
their use at the middle and high school levels is 
unknown. A version of the Motivated Strategies for 
Learning Questionnaire (MSLQ), developed for use 
with college samples, was adapted and used with 



middle school students. The High School Survey of 
Student Engagement (HSSSE) was modeled after the 
National Survey of Student Engagement (NSSE), a 
measure of engagement of college-age students. 

All but one measure (the Attitudes Towards 
Mathematics Survey [ATM]) has been used with at 
least one ethnically or economically diverse sample 
of students (see appendix A for information on 
populations). Other than the work conducted by 
the developer, information could not be found on 
the use of five of the measures (4-H Study for Posi- 
tive Youth Development School Engagement Scale, 
Consortium on Chicago School Research/Academic 
Engagement Scale [CCSR/AES], ATM, REI, and 
Student School Engagement Survey [SSES]). 



Student self-report questionnaires 

Student self-report measures can bring the critical 
voices and perspectives of students into school 
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TABLE 1 

Developer and availability of instruments 



Instrument 


Developer 


Availability/website 


Student self-reports 


4-H Study for Positive Youth 
Development: School Engagement 
Scale (4-H) 


Richard Lerner, Institute for Applied 
Research in Youth Development, Tufts 
University 


Available by contacting developer, at 
richardJerner@tufts.edu; http://ase.tufts. 
edu/iaryd 


Attitudes Towards Mathematics Survey 
(ATM) 


Raymond Miller, University of Oklahoma 


Available in Miller et al. (1996) 


Consortium on Chicago School 
Research/Academic Engagement Scale 
(CCSR/AES) 


Consortium on Chicago School Research 
(CCSR) 


http://ccsr.uchicago.edu/ 

surveymeasures2007 


Engagement versus Disaffection with 
Learning (EvsD), student report 


Ellen Skinner, Portland State University 


www.pdx.edu/psy/ellen-skinner-1 


High School Survey of Student 
Engagement (HSSSE) 


Center for Evaluation and Education 
Policy, Indiana University 


www.indiana.edu/~ceep/hssse/ 


Identification with School 
Questionnaire (ISQ) 


Kristin (Voelkl) Finn, Canisius College 


Available in Voelkl (1996) 


Motivated Strategies for Learning 
Questionnaire (MSLQ) 


Paul Pintrich and Elisabeth DeGroot, 
National Center for Research to Improve 
Postsecondary Teaching and Learning, 
University of Michigan 


Middle school version available in 
Pintrich and DeGroot (1990) 


Motivation and Engagement Scale 
(MES) 


Andrew Martin, Lifelong Achievement 
Group 


www.lifelongachievement.com 


Research Assessment Package for 
Schools (RAPS), student report 


Institute for Research and Reform in 
Education (IRRE) 


Available in RAPS manual (www.irre.org/ 
publications/) 


School Engagement Measure (SEM)- 
MacArthur 


Phyllis Blumenfeld and Jennifer Fredricks, 
MacArthur Network for Successful 
Pathways through Middle Childhood 


Available in Fredricks et al. (2005) 
or by contacting co-developer, at 
jfred@conncoll.edu 


School Engagement Scale/ 
Questionnaire (SEQ) 


Sanford Dornbusch, Stanford University, 
and Laurence Steinberg, Temple 
University 


Available by contacting co-developer, at 
lds@temple.edu 


School Success Profile (SSP) 


Gary Bowen and Jack Rickman, Jordan 
Institute for Families, University of North 
Carolina at Chapel Hill 


www.schoolsuccessprofile.org 


Student Engagement Instrument (SEI) 


James Appleton, Gwinnett County 
Schools, Georgia, and Sandy Christenson, 
University of Minnesota 


Available in Appleton et al. (2006) 
or by contacting developer, at 
Jim_Appleton@Gwinnett.k12.ga.us 


Student School Engagement Survey 
(SSES) 


National Center for School Engagement 
(NCSE) 


www.schoolengagement.org 


Teacher reports 


Engagement versus Disaffection with 
Learning (EvsD), teacher report 


Ellen Skinner, Portland State University 


www.pdx.edu/psy/ellen-skinner-1 


Reading Engagement Index (REI) 


Allan Wigfield and John Guthrie, 
University of Maryland 


Available in Wigfield et al. (2008) or by 
contacting developers, at aw44@umail. 
umd.edu orjg76@umail.umd.edu 


Research Assessment Package for 
Schools (RAPS), teacher report 


Institute for Research and Reform in 
Education (IRRE) 


Available in RAPS manual (www.irre.org/ 
publications/) 


Observational measures 


Behavioral Observation of Students in 
Schools (BOSS) 


Edward Shapiro, Lehigh University 


Manual can be ordered through Guilford 
Press (Shapiro 2004) 



(CONTINUED) 




WHAT INSTRUMENTS ARE AVAILABLE FOR MEASURING STUDENT ENGAGEMENT? 



7 



TABLE 1 (CONTINUED) 

Developer and availability of instruments 



1 Instrument 


Developer 


Availability/website 


Classroom AIMS 


Alysia Roehrig, Florida State University 


Available by contacting developer, at 
aroehrig@fsu.edu 


Code for Instructional Structure and 
Student Academic Response (MS- 
CISSAR) 


Charles Greenwood, Juniper Gardens 
Children's Project, University of Kansas 


www.jgcp.ku.edu/~jgcp/products/ 

EBASS/ebass_materials.htm 


Instructional Practices Inventory (IPI) 


Jerry Valentine, Middle Level Leadership 
Center, University of Missouri 


www.mllc.org 



Note: The Academic Engagement Scale has been translated into Polish and Spanish. The Motivated Strategies for Learning Questionnaire has been 
translated into nine languages. The School Success Profile is available in Spanish; parts of it have been translated into Hebrew, Lithuanian, Portuguese, and 
Romanian. The SEM-MacArthur has been translated into Spanish. 

Source: Authors' analysis of instrument documentation. 



reform and improvement efforts. The 14 self-report 
measures range from a 4-item scale (the CCSR / 
AES) to a 121-item questionnaire (the HSSSE). In 
some cases, the engagement items are a subset of a 
longer instrument that measures other constructs 
as well; some instrument names (for example, the 
ATM and the School Success Profile [SSP]) reflect 
their broader focus. It is up to users to determine 
whether a subset of engagement items from a 
larger instrument can be used for their purposes. 
However, care should be taken in using a sub- 
scale or set of items from a larger instrument or 
adapting scales by eliminating or changing items, 
because such changes may affect the instrument’s 
reliability and validity. Instrument developers or 
other experts should be consulted to identify im- 
plications of using subscales from a larger instru- 
ment or making changes to an item set. 

Copies of 11 of the 14 instruments are available at 
no cost in a published source, accessible online, or 
available by contacting the developer. The other 
three instruments (the SSP, the HSSSE, and the 
Motivation and Engagement Scale [MES]) must be 
purchased. The cost covers questionnaire mate- 
rials, survey administration, data preparation, 
preparation of individual and school reports, and 
other technical assistance. 

Most student self-report measures were admin- 
istered in classrooms. Because the engagement 
scales were sometimes part of a larger item set, 



the time to administer the subscales was gener- 
ally unknown. Several studies mentioned having 
an individual other than the teacher administer 
the questionnaire, to encourage students to be 
more honest in their reporting. Others mentioned 
the importance of reading all the items aloud 
to students, at least at the upper elementary to 
middle school level, to eliminate the possibility 
that students misread questions. 

One instrument, the SSP, trains registered users 
online. Some developers provide administration 
guidelines. Developers acknowledge the impor- 
tance of clear instructions so that survey admin- 
istration can be standardized. For example, the 
developers of the Research Assessment Package 
for Schools (RAPS) suggest that training for data 
collectors should include information on stan- 
dard instructions, pacing, maintaining focus, and 
answering questions. 



Teacher reports on students 

Three instruments (ranging from 3 to 20 items) 
involve teacher reports on individual student 
engagement. All three teacher report instruments 
are available at no cost. 

Two developers of student self-report measures 
(the EvsD and RAPS) also offer an instrument for 
teacher reports on student engagement. In the 
EvsD teacher report instrument, teachers complete 
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20 items on behavioral and emotional engagement 
for each student in their class. In the RAPS teacher 
report measure, teachers complete three items 
on each student. Neither teacher report is subject 
specific; both can be used in any subject area. In 
the third measure, the Reading Engagement Index 
(REI), teachers rate students on aspects of engaged 
reading, with ratings summed across the eight 
items for a total score. Teachers in one study com- 
pleted the REI in a 20 -minute session, suggesting 
that the rating time per 25 students in a classroom 
is less than a minute per student. 

For meaningful results, teachers should have ex- 
perience with the students before completing the 
items. Teacher ratings should be completed at the 
same time and in a consistent manner across all 
teachers in a study. 



Observational measures 



Four measures use observational methods to 
collect data on engagement. Two (the Behavioral 
Observation of Students in Schools [BOSS] and 
the MS-CISSAR) observe individual students; 
two others (Classrooms AIMS and the Instruc- 
tional Practices Inventory [IPI]) involve classroom 
observations. For all four, the developers stress 
the importance of well trained observers (that is, 
observers who have demonstrated that their obser- 
vation results are consistent with the results from 
a prerecorded criterion observation or with the 
observations of other trained observers). 



Four measures use 
observational methods 
to collect data on 
engagement — two 
observe individual 
students, and two 
involve classroom 
observations 



Student-level observations. BOSS and the MS-CIS- 
SAR assess students’ on- and off-task behavior in 
an instructional setting. Both involve systematic 
direct observations of students in 
classrooms using a standardized 
observation protocol to collect 
data on a specific, predetermined 
set of behaviors. These measures 
use a form of momentary time 
sampling, in which an observer 
records whether a student exhib- 
its a predetermined category of 
behavior during a defined interval. 



Data are reported as the percentage of occurrences 
of the observed behaviors out of the total number 
of observations. 

Determining the number of observations needed 
to get an accurate picture of a student is critical. 
The developer of BOSS recommends collecting 
data at multiple times and acknowledges that 
observers may need to collect data across aca- 
demic settings (group work, seat work, and so 
forth). The developer suggests three observations 
of 20-30 minutes each over 2-3 days. According 
to the developer, about 10-15 hours of training is 
required to become proficient at administering the 
measure. 

For the MS-CISSAR, observation data are recorded 
in 20 -second intervals, with the user determin- 
ing the length or total time a student is observed. 
Training is available through videotapes and other 
self-practice manuals. 

The observational codes of BOSS and MS-CISSAR 
instruments are publicly available in journal 
articles and books. The software systems and 
observer training must be purchased. 

Classroom observations. Two observational mea- 
sures, the Classroom AIMS and the IPI, focus on 
the classroom rather than the student. Classroom 
AIMS covers four areas: three categories of teach- 
ing practice (atmosphere, instruction/content, 
and management) and one category of student 
outcomes called engagement (four items). The 
four engagement items (constituting a subscale) 
are part of a larger set of 75 items that an observer 
completes on a teacher’s classrooms to assess the 
teacher’s use of effective teaching practices and 
success in maintaining high levels of observed 
student engagement. Studies using this measure 
have reported classroom observations of one to 
four hours occurring two to five times a year. The 
75 items are available from the developer, but no 
training is available. 

The IPI aggregates classroom observations 
(100-120 three-minute classroom observations per 
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school) to the school level. The developer recom- 
mends that schools collect data several times 
a year. The results are provided as percentages 
of classrooms falling into each of six categories 
of engaged learning. The schoolwide results are 
examined without reference to individual teachers, 
as the results are intended for use in faculty dis- 
cussions about schoolwide improvement of teach- 
ing and learning. The IPI is publicly available, but 
the developer does not recommend its use without 
training. Training is available in a one-day work- 
shop provided by the developers. Because the IPI is 
intended as a formative tool for faculty reflection 
on student-engaged learning in the school, the 
developers suggest that school administrators not 
be observers. 



WHAT ARE THE CHARACTERISTICS 
OF EACH IDENTIFIED MEASURE? 

The information in the instrument abstracts in ap- 
pendix A is summarized below to provide a broad 
overview of the characteristics of the identified 
measures. The information is organized into three 
sections that represent the kinds of questions 
someone searching for measures might have: 

• Definition of engagement. The “what is mea- 
sured” row of the instrument abstracts de- 
scribes how the instrument measures engage- 
ment (subscale names, sample items, number 
of items, and so forth). Substantial variation 
exists in how engagement is defined. One 
aspect of what is measured has to do with the 
dimensions of engagement assessed (behav- 
ioral, emotional, and cognitive). A second has 
to do with the object of engagement (engage- 
ment in school or engagement of all students 
or individual students in a classroom). 

• Purposes and uses. The “background” and 
“use” rows address why the instrument was 
developed and how it has been used. The 
purposes and uses are important because they 
help potential users understand how particular 
measures may align with their intended uses. 



The information on 
instruments is organized 
by the three kinds of 
questions someone 
searching for measures 
might have: definition 
of engagement, 
purposes and uses, and 
psychometric properties 



Technical informa- 
tion on the psycho- 
metric properties of 
the measure. Psycho- 
metric 1 properties 
refer to the descrip- 
tion of information 
gathered during the 
construction and 
validation of mea- 
sures that shows the 

degree to which the instrument is operating 
as intended (that is, how much evidence is 
available to support the appropriateness of 
inferences made as a result of employing the 
measure). Two important types of psychomet- 
ric information for potential users to consider 
are reliability and validity, detailed for each 
instrument in appendix A. The psychometric 
information provided is that found by the 
research team but may not be all the informa- 
tion available on a measure. An exhaustive 
search and review of the technical quality of 
individual instruments was not conducted, 
and judgments were not made about the 
quality of the studies cited or the adequacy 
of the technical information reported. Once a 
particular instrument use is identified, users 
should explore reliability and validity with 
developers or other experts in more depth 
relative to the intended use. 



Definition of engagement 

Developers use a broad range of terms to describe 
their instruments (student engagement , school 
engagement , academic engagement , engaged time , 
student engaged learning , academic responding , 
engagement in class, engagement in school work), 
illustrating the lack of commonly accepted termi- 
nology in this area. The dimensions and focuses of 
engagement also vary across instruments (table 2). 

Dimensions of engagement assessed. Several sum- 
maries of research on engagement (Fredricks, 
Blumenfeld, and Paris 2004; Jimerson, Campos, 
and Greif 2003; National Research Council and 
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Different instruments 
measure different 
types of engagement — 
behavioral, emotional, 
or cognitive 



Institute of Medicine 2004) 
describe it as having multiple 
dimensions. For example, En- 
gaging Schools: Fostering High 
School Students' Motivation to 
Learn (National Research Council 
and Institute of Medicine 2004) 
describes engagement in schoolwork as involving 
behaviors (persistence, effort, attention, taking 
challenging classes), emotions (interest, pride in 
success), and mental or cognitive aspects (solving 
problems, using metacognitive strategies). It also 
distinguishes between academic engagement and 
social engagement (participation in extracurricu- 
lar activities, having friends at school). 



Different instruments measure different types of 
engagement (behavioral, emotional, or cognitive; 
see table 2). When available, the developer’s lan- 
guage is used to check the dimensions of engage- 
ment assessed. Where the developer did not use 
the terms behavioral emotional , or cognitive , the 
instrument content was reviewed against typical 
descriptions of each dimension in the literature. 



Student self-report measures. Of the 14 student 
self-report instruments, 5 include subscales that 
address all three dimensions of engagement; 5 ad- 
dress two dimensions; and 4 address one dimen- 
sion (see table 2). 



• Behavioral engagement. Of the 14 student 
self-report measures, 11 include a focus on 
behavioral engagement; 4 have subscales 
entitled behavioral engagement (see table Cl 
in appendix C). Across measures, individual 
items ask students to report on their attention, 
attendance, time on homework, preparation 
for class, participation in class, concentration, 
participation in school-based activities, effort, 
persistence, adherence to classroom rules, and 
risk behaviors (such as skipping school). 

• Emotional engagement. Of the 14 student 
self-report measures, 10 include items having 
to do with emotional engagement; 5 include 
subscales labeled emotional engagement. 



Items address emotional reactions to school 
or aspects of school such as being happy 
or anxious, expressing interest and enjoy- 
ment, reporting fun and excitement, feeling 
safe, having supportive or positive relation- 
ships with teachers and peers, having family 
support for learning, expressing feelings of 
belonging, and valuing school. 

• Cognitive engagement. Of the 14 student self- 
report measures, 8 include items focusing 
on cognitive engagement; 3 have subscales 
labeled cognitive engagement. Two instru- 
ments (the ATM and the MSLQ) include items 
assessing self-regulation, defined as a set of 
metacognitive, motivational, and behavioral 
techniques a learner can use to manage and 
control learning processes; and aspects of cog- 
nitive strategy use, which include questions 
about the use of shallow or deep strategies to 
learn, remember, and understand material. 2 
Some instruments (the Student Engagement 
Instrument [SEI] and the Student School En- 
gagement Survey [SSES]) ask students about 
the importance of schooling, learning goals, 
or future aspirations as an aspect of cognitive 
engagement. 

Items used to measure behavioral, emotional, and 
cognitive engagement are sometimes used incon- 
sistently across instruments. For example, some 
instruments consider the extent of participation 
in class as an aspect of behavioral engagement, 
whereas others consider it an aspect of cognitive 
engagement. Some instruments use student effort 
to describe the degree of psychological investment 
in learning (cognitive engagement), whereas others 
use it to reflect compliance with the work required 
in school (behavioral engagement). Students’ valu- 
ing of school is used as part of both emotional and 
cognitive engagement measures. The Identification 
with School Questionnaire (ISQ), for example, 
assumes that valuing is emotional and reflects 
how much students value school as an important 
institution in society and as useful to achieving 
their future goals (Voelkl 1997). Other measures 
assume that valuing reflects cognitive engagement 
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TABLE 2 

Dimensions of engagement assessed by instruments 




Source: Authors' analysis of instrument documentation. 



(students’ beliefs and self- appraisals of their 
learning goals). (Table Cl in appendix C provides 
additional information on the student self-report 
measures, including the subscale names used 
and sample item wording. Table C2 shows the 
subscales, categorized by the three dimensions of 
engagement, used across student self-reports.) 

Teacher report measures. The three teacher report 
measures involve teacher ratings of individual 
student engagement. The EvsD (a 20 -item instru- 
ment) comprises four subscale scores reflecting 



both positive and negative aspects of behavioral 
and emotional engagement. The RAPS teacher 
report includes three items that assess both 
behavioral and emotional engagement, which are 
summed to yield a general measure of student 
engagement. The REI produces one total score 
from eight items intended to cover behavioral, 
emotional (motivational), and cognitive aspects of 
reading engagement. 

Observational measures. BOSS and the MS- 
CISSAR measure a targeted individual student’s 
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The choice of an 
instrument depends 
on the intended use; 
no single instrument is 
best for all purposes 



on- and off-task behavior or time 
engaged in classroom settings. 

As such, they focus on categories 
of observed behavioral engage- 
ment. Classroom AIMS includes 
an engagement subscale with 
four items assessing observ- 
able aspects of classroom behavioral engage- 
ment (whether most students stay on task) and 
emotional engagement (whether students are 
excited about content). The IPI measures student- 
engaged learning (that is, the extent of higher- 
order/deep learning in classrooms), which is 
similar to cognitive engagement. 



Measures of engagement in school or class. The 

instruments studied assess general engagement 
in school or engagement in a particular class (for 
example, in content or subject areas; table 3). Nine 
of the student self-report measures include items 
worded to reflect general engagement in school. 
Five (the CSSR/AES, ATM, EvsD, MSLQ, and the 
School Engagement Scale/Questionnaire [SEQ]) 
are worded for use at the class level. The CCSR / 
AES is administered in language arts classes. The 
ATM, MSLQ, and SEQ have been used in vari- 
ous high school subject areas. The EvsD assesses 
engagement in the classroom in general. 



Two teacher report instruments (the EvsD and 
RAPS) are also class specific. Teachers rate 
students based on their knowledge of students in 
their own classroom context. The third teacher 
report instrument, the REI, assesses students as 
engaged readers in a particular class context. 



Purpose and uses 

The instruments are used for a variety of purposes. 
The classification in table 4 is not intended to sug- 
gest that each instrument should be used only for 
the identified purposes but to show potential users 
how the instruments have been used in previous 
studies. 

The choice of an instrument depends on the 
intended use; no single instrument is best for 



all purposes. The need to compare results with 
normative data available from the developer, for 
example, limits the choices to instruments such 
as the HSSSE, SSP, MES, RAPS, and IPI, which 
have such comparison data available to help in 
interpreting the results. District or community 
organizations looking for a broad-based survey 
to compare aspects of adolescents’ well-being for 
use in needs assessment discussions can choose 
between just 2 of the 21 instruments (4-El, SSP). 

A school psychologist who wants to observe a stu- 
dent over time and track observed engagement to 
see whether a particular intervention seems to be 
helping can also choose between just two instru- 
ments in the set (BOSS, MS-CISSAR). Schools 
interested in tracking increases in student engage- 
ment over time as a school improvement goal must 
determine whether their interest is in engagement 
in school; engagement at the class level, in particu- 
lar subjects; or particular skills, such as reading, 
because different measures assess engagement in 
different contexts. They must also consider the 
potential usefulness of including multiple mea- 
sures of engagement, comparing and contrasting 
data from students, teachers, and observational 
methods to better understand the current state of 
student engagement. 

Research on student motivation and cognition. 

Several measures were developed by research 
psychologists studying motivation, cognition, and 
engagement. The EvsD student and teacher report 
instruments were developed in the early 1990s, 
through research testing a theory of motiva- 
tion linking aspects of the environment (such as 
the degree to which the teacher makes students 
feel as if they belong) with patterns of student 
behavior (such as extent of student engagement) 
and achievement outcomes (Connell 1990). In 
1998, Connell and others at the IRRE revised the 
original instruments to create a shorter set of 
instruments (RAPS) for evaluating school reform 
efforts based on the same theoretical framework. 
Two other measures (the ATM and the MSLQ) 
were developed as part of research exploring the 
relationships among students’ self-regulation in 
learning, cognitive strategy use, and achievement 
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TABLE 3 

Instruments with sample items by school or class focus 









Setting 


Instrument 


Sample item 


In school In class 


Student self-reports 


4-H Study for Positive Youth Development: 
School Engagement Scale (4-H) 


1 want to learn as much as 1 can in school. 


/ 




Academic Engagement Scale (CCSR/AES) 
(administered in language arts classes in 
Chicago) 


1 work hard to do my best in this class. 




/ 


Attitudes Towards Mathematics Survey (ATM) 


If 1 have trouble understanding a problem, 1 go over 
it again until 1 understand it. 




/ 


Engagement versus Disaffection with 
Learning (EvsD), student report 3 


When I'm in class, 1 listen very carefully. 




/ 


High School Survey of Student Engagement 
(HSSSE) 


How do you feel about the following statements 
related to your high school? 


/ 




Identification with School Questionnaire (ISQ) 


School is one of my favorite places to be. 


/ 




Motivated Strategies for Learning 
Questionnaire (MSLQ) 


1 outline the chapters in my book to help me study. 




/ 


Motivation and Engagement Scale (MES) 


I've given up being interested in school. 


/ 




Research Assessment Package for Schools 
(RAPS), student report 3 


1 work hard on my schoolwork. 


/ 




School Engagement Measure (SEM)- 
MacArthur 


1 am interested in the work at school. 


/ 




School Engagement Scale/Questionnaire 
(SEQ) 


How often does your mind wander in each of these 
classes? 




/ 


School Success Profile (SSP) 


1 find school fun and exciting. 


/ 




Student Engagement Instrument (SEI) 


Learning is fun because 1 get better at something. 


/ 




Student School Engagement Survey (SSES) 


1 feel excited by the work in school. 


/ 




Teacher reports 


Engagement versus Disaffection with 
Learning (EvsD), teacher report 3 


In my class, this student does more than required. 




/ 


Reading Engagement Index (REI) 


This student works hard in reading. 




/ 


Research Assessment Package for Schools 
(RAPS), teacher report 3 


In my class, this student seems tuned in. 




/ 


Observational measures 


Behavioral Observation of Students in Schools 
(BOSS) 


Observations are coded using five categories (active 
engagement, passive engagement, off-task motor, 
off-task verbal, and off-task passive). 




/ 

(focus on 
individual 
student) 


Classroom AIMS 


Observers respond to four items about 
engagement levels in the class (for example, at least 
80 percent of students are consistently on task and 
highly engaged in class activities). 




/ 

(classroom 

focus) 


Code for Instructional Structure and Student 
Academic Response-Mainstream Version (MS- 
CISSAR) 


Observations of student behavior are coded using 
three categories (positive engagement behaviors, 
neutral engagement behaviors, and inappropriate 
behaviors). 




/ 

(focus on 
individual 
student) 


Instructional Practices Inventory (IPI) 


Observations of classrooms are coded using a six- 
level rubric of extent of engaged student learning. 




/ 

(classroom 

focus) 



a. Includes separate student self-report and teacher report instruments. 
Source: Authors' analysis of instrument documentation. 
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TABLE 4 

Purposes and uses of instruments 



Instrument 


Research on 
motivation 
and cognition 


Research on 
dropping out 


Evaluation of 
interventions 


Monitoring at 
the teacher, 
school, or 
district level 


Diagnosis and 
monitoring at 
the student 
level 


Needs 

assessment 


Student self-reports 


4-H Study for Positive Youth 
Development: School 
Engagement (4-H) 






/ 

(4-H 

participation) 






/ 


Academic Engagement Scale 
(CCSR/AES) 








/ 






Attitudes Towards Mathematics 
Survey (ATM) 


/ 












Engagement versus Disaffection 
with Learning (EvsD), student 
report 3 


/ 












High School Survey of Student 
Engagement (HSSSE) 








/ 






Identification with School 
Questionnaire (ISQ) 




/ 


/ 

(class size; 
magnet 
schools) 








Motivated Strategies for 
Learning Questionnaire (MSLQ) 


/ 




/ 

(instructional 

strategies) 








Motivation and Engagement 
Scale (MES) 


/ 




/ 

(youth 

enrichment 

program) 


/ 


/ 




Research Assessment Package 
for Schools (RAPS), student 
report 3 


/ 




/ 

(school 

reform) 


/ 


/ 




School Engagement Measure 
(SEM)-MacArthur 


/ 












School Engagement Scale/ 

Questionnaire 

(SEQ) 


/ 




/ 

(instructional 

strategies) 








School Success Profile (SSP) 






/ 

(social 

supports) 


/ 




/ 


Student Engagement Instrument 
(SEI) 




/ 


/ 

(dropout 

prevention) 


/ 


/ 




Student School Engagement 
Survey (SSES) 






/ 

(truancy 

reduction) 








Teacher reports 


Engagement versus Disaffection 
with Learning (EvsD), teacher 
report 3 


/ 













(CONTINUED) 
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TABLE 4 (CONTINUED) 

Purposes and uses of instruments 



Instrument 


Research on 
motivation 
and cognition 


Research on 
dropping out 


Evaluation of 
interventions 


Monitoring at 
the teacher, 
school, or 
district level 


Diagnosis and 
monitoring at 
the student Needs 

level assessment 


Reading Engagement Index (REI) 


/ 

(reading 

motivation) 




/ 

(professional 

development) 






Research Assessment Package 
for Schools (RAPS), teacher 
report 3 


/ 




/ 

(school 

reform) 






Observational measures 


Behavioral Observation of 
Students in Schools (BOSS) 






/ 

(remediation) 




/ 


Classroom AIMS 






/ 

(teacher 

mentoring) 


/ 




Code for Instructional Structure 
and Student Academic 
Response-Mainstream Version 
(MS-CISSAR) 






/ 

(instructional 

strategies) 




/ 


Instructional Practices Inventory 

(IPI) 






/ 

(school 

reform) 


/ 





a. Includes separate student self-report and teacher report instruments. 
Source: Authors' analysis of instrument documentation. 



outcomes. Research in this area examines the use 
of cognitive, metacognitive, and self-regulatory 
strategies that foster active engagement in learning 
(Corno and Mandinach 1983; Meece, Blumenfeld, 
and Hoyle 1988). 

Research on dropping out. A long line of research 
explores disengagement as a precursor to drop- 
ping out (Appleton, Christenson, and Furlong 
2008; Finn 1989). Two measures were developed 
by researchers investigating this issue— the ISQ 
and the SEI. The ISQ was developed to assess how 
much students identify with or disengage from 
school, based on the hypothesis that identifying 
with ones school is crucial in preventing dropouts 
(Finn 1989). The SEI was developed to go beyond 
observable indicators of academic and behavioral 
engagement (time on task, attendance, homework 
completion) and measure the cognitive and psy- 
chological aspects of engagement as reported by 
students themselves. 



Evaluation of interventions. Many measures have 
been used to study the effects of interventions or 
school reform efforts on increasing engagement or 
reducing aspects of disengagement (dropout rates, 
truancy). RAPS was developed for use in schools 
implementing First Things First, a school reform 
model aimed at promoting student engagement 
and learning. The items used on the SSES were 
compiled from other pre-existing measures of 
engagement by the NCSE for evaluating interven- 
tions aimed at reducing truancy. 

Monitoring of engagement at the teacher ; school , 
or district level. Some measures have been used 
to inform improvement efforts at the teacher, 
school, or district level based on the assumption 
that student engagement is important to monitor. 
Two student self-report measures (the CCSR/AES 
and the HSSSE) provide feedback to schools on 
their students’ engagement, which can be com- 
pared with the results for other schools or national 
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This report summarizes 
evidence on two 
broad categories 
of psychometric 
information: reliability 
and validity — both of 
which are multifaceted 



norms. The Chicago Consortium 
on School Research converts 
scores from the CCSR/AES to 
qualitative descriptors that reflect 
the level of risk. The scale results 
classify students into four catego- 
ries (none, limited, moderate, or 
high engagement). Schools can 
then examine the percentage of 
students in each category across time or compare 
percentages with those of other schools. Schools 
that participate in the HSSSE receive customized 
reports that compare their students’ responses 
with those of other schools. Classroom AIMS has 
been used to provide feedback to teachers on their 
use of instructional strategies and levels of student 
engagement. The IPI aggregates classroom obser- 
vations to provide formative data to school faculty 
on the extent of student-engaged learning. 



Diagnosis and monitoring of students at risk for 
disengagement. Other measures were developed 
to identify and assess students at risk for disen- 
gagement and academic failure to provide bet- 
ter services to these students. The MES student 
self-report measure creates individual profiles 
across 11 subscales reflecting a multidimensional 
model of motivation and engagement. Users of this 
measure are given a list of targeted interventions 
for students who match particular profiles of low 
motivation and engagement. Two observational 
systems, BOSS and the MS-CISSAR, respond to 
the need of school psychologists and others for 
standardized observational measures of student 
behavior to supplement achievement measures. 
These measures have been used to assess indi- 
vidual students in both typical and special needs 
populations, especially students at risk for aca- 
demic failure. 



Needs assessment. Two student self-report mea- 
sures (the 4-H and the SSP) were developed as 
part of larger projects to collect survey data on the 
percentage of youths in a community with posi- 
tive developmental assets. (Developmental assets 
are the positive experiences and qualities that 
can prevent high-risk behavior, enhance positive 



outcomes, and contribute to resiliency.) The items 
used in the 4-H study were initially part of the 
Profiles of Student Life: Attitudes and Behaviors 
Survey (PSL-AB), which measures 40 assets (posi- 
tive experiences and qualities) considered impor- 
tant in positive youth development. The school 
engagement scale of the SSP is one of 22 core 
dimensions assessed as part of a larger instrument 
intended to promote academic performance and 
close the achievement gap. 



Technical information available on the 
psychometric properties of measures 

A key aspect to consider in selecting a measure of 
engagement is its psychometric properties. There 
are many ways to collect and report empirical 
evidence about how scores from an instrument 
behave and whether they behave as intended. 

This report summarizes evidence on two broad 
categories of psychometric information: reliability 
and validity (both of which are multifaceted). The 
report describes evidence found on three types 
of reliability and two types of validity (table 5). 

(For information on definitions and methods for 
examining reliability and validity, see AERA/APA/ 
NCME 1999; Crocker and Algina 1986). 

Reliability refers to the degree to which an instru- 
ment produces consistent results. For example, 
a measure should yield consistent results across 
administrations. A measuring technique is unreli- 
able, for example, if a scale registers a person’s 
weight at 140 one day and 110 the next. 

Reliability is necessary but not sufficient: a 
measure can be highly reliable but not valid. The 
validity of an instrument reflects whether the re- 
sults obtained from using the instrument actually 
measure what was intended and not something 
else. Validity is concerned with the appropriate- 
ness of inferences made as a result of employing 
a measure. Instruments are validated to provide 
evidence to support inferences about the construct 
of interest. According to Hopkins, Stanley, and 
Hopkins (1990), if a measure lacks validity, the 
information provided is useless; the validity of a 
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TABLE 5 

Reliability and validity information reported 







Reliability 




Validity 


Instrument 


Internal 

consistency 

(Cronbach's 

alpha) 


Test-retest 

correlation 


Interrater 

agreement 


Construct 


Criterion- 

related 


Student self-reports and teacher reports 


4-H Study for Positive Youth Development: 
School Engagement Scale (4-H) 


.63-90 











/ 


Consortium on Chicago School Research/ 
Academic Engagement Scale (CCSR/AES) 


.65-68 








/ 


/ 


Attitudes Towards Mathematics Survey (ATM) 


.63-81 


— 


— 


/ 


/ 


Engagement versus Disaffection with Learning 
(EvsD), student and teacher reports 


.61 -.85 
student 
report, .81- 
.87 teacher 
report 


.53-.68 
student 
report, .65- 
.82 teacher 
report 




/ 


/ 


High School Survey of Student Engagement 
(HSSSE) 


_ 


_ 


_ 


_ 


_ 


Identification with School Questionnaire (ISQ) 


.54-.84 


— 


— 


/ 


/ 


Motivated Strategies for Learning 
Questionnaire (MSLQ) 


.63-88 


_ 


_ 


/ 


/ 


Motivation and Engagement Scale (MES) 


.70-.87 


.61 -.81 


— 


/ 


/ 


Reading Engagement Index (REI) 


.89-.93 


— 


— 


/ 


/ 


Research Assessment Package for Schools 
(RAPS), student and teacher reports 


.68-.77 
student 
report, 
.87 teacher 
report 






/ 


/ 


School Engagement Measure (SEM)-MacArthur 


.55-.86 


— 


— 


/ 


/ 


School Engagement Scale/Questionnaire (SEQ) 


.74-.86 


— 


— 


/ 


/ 


School Success Profile (SSP) 


.66-.82 


— 


— 


/ 


/ 


Student Engagement Instrument (SEI) 


J2-.92 


— 


— 


/ 


/ 


Student School Engagement Survey (SSES) 


A9-.92 


— 


— 


/ 


/ 


Observational measures 


Behavioral Observation of Students in Schools 
(BOSS) 








.90-1.00 


/ 


/ 


Classroom AIMS 


.62-79 


— 


.58-.65 


/ 


— 


Code for Instructional Structure and Student 
Academic Response — Mainstream Version 
(MS-CISSAR) 






.80-.97 




/ 


Instructional Practices Inventory (1 PI) 


— 


— 


00 

0 

1 

VO 

o 


/ 


/ 



— is not available. 

Note: Ranges within cells indicate results for different subscales, age groups, or results by different researchers. Checkmarks indicate that information on 
validity is available, not that measures are necessarily valid. 

Source: Authors' analysis of instrument documentation. 
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The internal consistency 
of the student and 
teacher report 
engagement scales/ 
subscales as reported 
by the instrument 
developers or others 
ranged from .49 to 
.93, with most scales 
inthe.70-.80 range 



measure can be described in terms 
of the “accuracy of specified infer- 
ences made from its scores” (p. 

76). Validity has to be examined 
in the context of a specific use of 
a measure. For example, cranial 
measurements can be obtained 
reliably and are valid measures 
for use in determining head size, 
but they are not a valid measure of 
intelligence (Leary 2004). 



The instrument abstracts in 
appendix A briefly describe the information on 
reliability and validity identified from the refer- 
ences on each instrument. The research team did 
not conduct an exhaustive search and review of 
the technical quality of the instruments, so no 
judgments are made about the quality of the stud- 
ies cited or the adequacy of the technical informa- 
tion reported. Potential users should explore the 
information available on reliability and validity 
in more depth before selecting a measure for a 
particular use. 



Reliability. Three types of reliability are summa- 
rized: internal consistency, test-retest correlation, 
and interrater agreement. 



Internal consistency. Internal consistency is the 
extent to which individuals who respond one way 
to one item respond the same way to other items 
intended to measure the same thing. This kind of 
reliability is important for measures that sum or 
average responses across a set of items to provide a 
single scale score. 



Internal consistency can be examined using 
Cronbach’s alpha coefficient, which results in 
values ranging from 0 to 1. The closer to 1, the 
more internally consistent a set of items is for that 
sample of respondents. A Cronbach s alpha of .70 
or higher for a set of items is considered acceptable 
(Leary 2004). 



The internal consistency of the student and teacher 
report engagement scales/subscales as reported by 



the instrument developers or others ranged from 
.49 to .93, with most scales falling in the range of 
.70-80 (see table 5). 3 The lowest alpha (.49) was 
reported for the behavioral engagement subscale 
of the SSES. The low reliability of the behavioral 
engagement scale was present at only one of the 
three intervention sites, however, with alphas of 
.79 and .80 at the other two. 

The alpha coefficient of the SEM-MacArthur 
cognitive engagement subscale was .55 during the 
first wave of data collection. The developers made 
several changes in the wording of the cognitive en- 
gagement items and excluded students in grade 3, 
who had difficulty answering the questions. These 
changes raised the alpha to .82 in the second wave 
of data collection. 

As these examples suggest, the range of alphas can 
be broad. It is important to examine the informa- 
tion on reliability closely in light of the sample and 
intended use of the measure selected. 

Test-retest correlation. Test-retest reliability is the 
degree to which two different administrations of 
the measure give the same results. Correlations 
obtained can range from 0 to 1. The closer to 1, 
the greater the confidence that the instrument can 
produce consistent results. 

Few instruments reported information on re- 
peated administrations to the same group. The 
EvsD reported cross-year correlations of .53-68 
for the student self-report measure and .65-.82 for 
the teacher report. The MES developers reported 
test-retest reliability for each of the instrument 
subscales of .61 to .81 across two administra- 
tions several weeks apart. Test-retest correlations 
need to be examined in light of the length of time 
between the two administrations. (A two-week 
test-retest correlation should be higher than a 
cross-year correlation.) 

Interrater agreement. Interrater reliability is 
relevant for observational measures. It provides 
evidence of the consistency of ratings across two 
or more observers who use a prespecified coding 
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system. Training observers is critical for obtaining 
high interrater reliability. BOSS, MS-CISSAR, and 
IPI developers reported post-training interrater 
reliability at acceptable levels. This type of reliabil- 
ity can be reported as the percentage of times the 
observers agreed after observing the same behav- 
ior. Studies of the MS-CISSAR reported at least .80 
agreement between observers after training. 

Interrater reliability can also be reported in terms 
of the correlation between ratings by different 
observers. The developers of the IPI reported 
that interrater reliability was just .05-.20 among 
observers without training but at least .80 among 
a group of observers who attended the one-day 
training workshop. 

Validity. Many types of data can be collected to 
provide evidence of validity. The information 
reported for each instrument is described in ap- 
pendix A. 

Construct validity. Construct validity refers to the 
appropriateness of inferences made when using 
a measure. Information on construct validity is 
reported on all but three instruments (the 4H, 
the HSSSE, and the MS-CISSAR). Potential users 
should closely examine evidence provided by 
developers on how accurately a particular measure 
assesses a particular construct. 

Several types of correlational analyses (validity 
studies) were reported as evidence of construct va- 
lidity. The CCSR/AES was found to correlate posi- 
tively with student perceptions of teacher support. 
The RAPS engagement scale correlated as expected 
with students’ perceptions of aspects of their school 
environment. The student engagement scales in 
the SSP were reported to correlate positively with 
student perceptions of teachers, parents, and peer 
support. As Leary (2004, p. 71) notes: “No general 
criteria can be specified for evaluating the size of 
correlations when assessing construct validity. The 
size of each correlation coefficient must be consid- 
ered relative to the correlation we would expect to 
find if our measure were valid and measured what 
it was intended to measure.” 



Construct validity can also be informed by com- 
paring patterns of correlations across methods 
of measuring engagement. The developers of five 
instruments (BOSS, EvsD, RAPS, REI, and SEM- 
MacArthur) provided information on the correla- 
tions between data on engagement collected using 
different methods (student self-reports, teacher 
reports, and observations). For the EvsD, cor- 
relations between student and teacher measures 
were somewhat stronger for behavioral engage- 
ment than those for emotional engagement. This 
finding may reflect the fact that teachers can more 
easily observe student behaviors than student 
emotions. 



Nine instruments (the Classroom AIMS, EvsD, 
ISQ, MES, MSLQ, RAPS, REI, SEI, and SEM- 
MacArthur) reported results from either explor- 
atory or confirmatory factor analyses to support 
the validity of their subscales. (A factor analysis 
is a statistical technique that can be used to 
identify the underlying factors that account for 
relationships among items on a measure.) For 
a particular sample of respondents, responses 
to items measuring the same thing should load 
together on the same factor. For example, the 
developers of the ISQ used confirmatory factor 
analyses with 16 items to confirm two subscales 
(belonging and value); the developers of the MES 
used confirmatory factor analyses on a set of 
44 items, resulting in 11 subscales (self-belief, 
learning focus, valuing of school, persistence, 
planning, study management, disengagement, 
self-sabotage, anxiety, failure avoidance, and 
uncertain control). 



Criterion-related validity. 
Criterion-related validity 
refers to how well scores 
from a measure relate 
to key behaviors or 
outcomes. This type of 
validity is particularly 
important for poten- 
tial users to examine, 
because engagement is 
often considered a proxy 



Several types of 
correlational analyses 
were reported as 
evidence of construct 
validity, while 13 
instruments reported 
positive correlations with 
some measure of student 
achievement (criterion- 
related validity) 
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for student achievement (that is, if students are 
engaged in school, they are hypothesized to learn 
more over time and perform better on tests). 
Engagement scores would therefore be expected to 
correlate positively with grades or other mea- 
sures of academic performance; potential users of 
engagement measures should examine whether a 
particular measure is reported to have such a rela- 
tionship with relevant achievement measures. Ten 
of the student self-report measures of engagement 
(4-H, ATM, ISQ, MSLQ, MES, RAPS, SEM, SEQ, 
SEI, SSES), two of the teacher report instruments 
(RAPS, REI), and one observational measure (MS- 
CISSAR) reported positive correlations with some 
measure of student achievement. The developers 
of two instruments (RAPS and the SSES) reported 
positive correlations between their measures of 
student engagement and student attendance. 



STUDY LIMITATIONS 

Readers should keep in mind several caveats in 
reading this study: 

• The 21 instruments described may not include 
all engagement measures; some instruments 
might not have been found in the search, and 



some might have been excluded for reasons 
outlined in appendix B. The research team 
searched for instruments that used the term 
engagement or that were described as measur- 
ing engagement. The search did not include 
measures developed to assess closely related 
concepts that have not been described by ei- 
ther the developers or subsequent researchers 
as engagement measures. 

The abstracts in appendix A do not purport 
to cover the entire body of evidence on the 
use of each instrument. They provide only a 
useful starting point for understanding what 
is available. 

This study does not assess the quality of the 
studies or citations in the reference section of 
the abstracts of each instrument in appendix 
A. It describes what the studies reported in 
terms of reliability, validity, and uses without 
examining the quality of the studies them- 
selves. The inclusion of an instrument in 
this report should thus not be interpreted as 
a judgment about its quality or as advocacy 
of its use. It is left up to users to judge the 
technical quality of the instruments for their 
particular purposes. 
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NOTES 



1. Psychometrics is the field of study concerned 
with the theory and technique of educational 
and psychological measurement, including 
the construction and validity of measurement 
instruments such as questionnaires, tests, and 
personality assessments. 

2. Deep learning strategies help students under- 
stand material (using higher-order thinking, 
for example). Shallow strategies help students 



memorize or reproduce knowledge, with 
little attempt at deeper levels of analysis or 
understanding. 

3. One instrument, the HSSSE, used an item-by- 
item reporting format for several years but 
now groups the 121 items in the questionnaire 
by the three dimensions of engagement. No 
information on the internal consistency of the 
item groupings is available from the developer. 
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APPENDIX A 

INSTRUMENT ABSTRACTS 

This appendix includes abstracts of the 21 in- 
struments reviewed. (The student and teacher 
reports of the EvsD and RAPS are shown together, 



in tables A4 and A9, reducing the number of 
abstracts to 19.) Tables A1-A14 describe the 14 
student self-report instruments (and the EvsD and 
RAPS teacher reports). Table A15 describes a third 
teacher report instrument, the REI. Tables A16- 
A19 describe the four observational instruments. 



Student self-report instruments 



TABLE A1 

4-H Study for Positive Youth Development: School Engagement Scale (4-H) 



Feature 


Description 


Developer/website 


Developed by Dr. Richard Lerner and his colleagues at the Institute for Applied Research in Youth 
Development at Tufts University as part of the longitudinal 4-H Study of Positive Youth Development 
(http://ase.tufts.edu/iaryd). 


Availability/ 


A copy of the engagement scale items can be obtained by contacting Dr. Richard Lerner or Yibing Li 


key source 


at the Institute for Applied Research in Youth Development (http://ase.tufts.edu/iaryd). Engagement 
items were drawn from the Profiles of Student Life: Attitudes and Behaviors (PSL-AB) Survey, created by 
the Search Institute (Leffert et al. 1998), which can be used as a needs assessment tool and is available 
for purchase from the Search Institute (www.search-institute.org/survey-services/surveys/attitudes- 
behaviors). 


Population 


4-H study followed sample of grade 5 students from variety of racial/ethnic backgrounds in 13 states in 
rural, suburban, and urban areas through grade 11 (6,450 students from 34 states). (More than 2 million 
students in suburban, urban, and rural communities across the United States have responded to the PSL- 
AB Survey, developed for use with grades 6-12.) 


Method 


Student self-report questionnaire. 


Background 


Includes three subscales: behavioral school engagement, emotional school engagement, and cognitive 
engagement (the cognitive engagement subscale has not been included in a published study) (Lerner 
et al. 2008). Items included are part of a larger survey administered longitudinally to respondents with 
varying levels of involvement in community-based programs, such as 4-H, Boys and Girls Clubs, and 
YMCAs. They are adapted from items included in the PSL-AB. 


Administration 


Administered by: Trained members of the research staff in school or after-school program settings. 
Form: Paper and pencil questionnaire. 

Time: Scale is part of larger 4-H survey, which takes about two hours to administer. 
Training/instructions: Students were told that they could skip any question they did not want to answer. 


Languages 


English 


What is measured 


The engagement scale includes three subscales: 

• Behavioral engagement (five items): measures participation and effort (sample items: "1 complete 
homework on time," "I work hard to do well"). 

• Emotional engagement (five items): measures positive affect and school belonging (sample items: "1 
feel happy to be part of school," "1 enjoy the classes 1 am taking"). 

• Cognitive engagement (five items): measures school value (sample items: "1 want to learn as much as 1 
can at school," "School is important for future success"). 


Scoring/reporting 


Scale scores are obtained by reverse scoring negatively worded items, so that higher scores indicate 
higher engagement for all items. Response formats are 0 = usually to 3 = never for some items and 
0 = strongly agree to 3 = strongly disagree for others. 



(CONTINUED) 
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TABLE A1 (CONTINUED) 

4-H Study for Positive Youth Development: School Engagement Scale (4-H) 



Feature Description 



Reliability 


4-H developers report Cronbach's alphas for subscales of .70 (behavioral), .82 (emotional), and .90 
(cognitive). PSL-AB developers report Cronbach's alpha for four school engagement items of .63 (Leffert 
etal. 1998). 


Validity 


Li et al. (2008) report evidence of criterion-related validity through positive correlations between 4-H 
emotional and behavioral engagement subscales and achievement and through negative correlations 
between emotional and behavioral engagement subscales and risk behaviors and depression. 


Use 


4-H engagement scales have been used only by developers in correlational studies using the 4-H sample. 
For the PSL-AB, users are given a report on the percentage of students in their community (overall and by 
race and gender) that indicated each asset. (School engagement is one of 40 developmental assets.) 


References 


Jelicic, H., Bobek, D. L., Phelps, E., Lerner, R. M., and Lerner, J. V. (2007). Using positive youth development 
to predict contribution and risk behaviors in early adolescence: findings from the first two waves of the 
4-H study of positive youth development. International Journal of Behavior Development, 37(3), 263-273. 

Leffert, N., Benson, P. L., Scales, P. C., Sharma, A. R., Drake, D. R., and Blyth, D. A. (1998). Developmental 
assets: measurement and prediction of risk behaviors among adolescents. Applied Developmental Science, 
2(4), 209-230. 

Lerner, R. M., Lerner, J. V., Phelps, E., et al. (2008). The positive development of youth technical report: the 
4-H study of positive youth development Report of the findings from the first four waves of data collection. 
Retrieved October 1, 2009, from http://ase.tufts.edu/iaryd/documents/4HStudyAnnualReport2008.pdf. 

Li, Y., Bebiroglu, N., Phelps, E., Lerner, R. M., and Lerner, J. V. (2008). Out-of-school-time activity 
participation, school engagement and positive youth development: findings from the 4-H study 
of positive youth development. Journal of Youth Development, 3(3), doi: 080303FA001. http://data. 
memberclicks.com/site/nae4a/JYD_09080303_final. pdf. 

Theokas, C., Almerigi, J., Lerner, R. M., Dowling, E. M., Benson, P. L., Scales, P. C., and von Eye, A. (2005). 
Conceptualizing and modeling individual and ecological asset components of thriving in early 
adolescence. Journal of Early Adolescence, 25, 113-143. 



Source: Authors' compilation. 
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TABLE A2 

Academic Engagement Scale of the Consortium on Chicago School Research Biennial Survey (CCSR/AES) 



Feature Description 



| Feature 


Description 


Developer/website 


Developed by the CCSR (http://ccsr.uchicago.edu). 


Availability/ 
key source 


Information on the CCSR/AES, including items, can be found on developer's website (http://ccsr.uchicago. 
edu/surveymeasures2007/engg.html). Information on larger set of survey items used in the biennial 
survey can be found at http://ccsr.uchicago.edu/surveymeasures2007. Use of CCSR survey items is 
permitted without charge; however, CCSR must be cited as source of the items in documentation and 
publications. 


Population 


Since 1999, this scale, along with other scales and measures important to the CCSR, has been 
administered to more than 100,000 demographically diverse elementary and high school students 
attending Chicago Public Schools. 


Method 


Student self-report questionnaire. 


Background 


The AES examines students' reports about their interest and engagement in learning in their reading 
or language arts courses. Questions ask about students' interest in topics they are studying and their 
engagement in reading or language arts classroom in general. 

First developed and administered by Chicago Public Schools in 1999, this instrument has been 
administered biennially as part of the its regular survey collection schedule. The survey is administered 
only in reading and language arts courses. Purpose of survey is to monitor how school district as a whole 
and individual schools are performing in areas assessed. 


Administration 


Administered by: Teachers, during reading or language arts courses. 

Form: Paper and pencil questionnaire. 

Time: Administered as part of larger survey conducted in Chicago, usually within a class period. 

Training/instructions: Examples of survey forms used by CCSR, which include a sample cover page and 
student instructions, are available on CCSR website. 


Languages 


Multiple (English, Polish, and Spanish; to obtain scales in other languages, contact CCSR) 


What is measured 


Scale contains six items intended to provide general measure of academic engagement. 

Sample items include "1 work hard to do my best in this class" and "Sometimes 1 get so interested in my 
work 1 don't want to stop." 


Scoring/reporting 


Four-point response scale ranges from 1 (strongly disagree) to 4 (strongly agree). Scale scores can be 
developed by averaging responses to six items (after reverse scoring negatively worded items). Higher 
scores indicate that students report higher engagement. Based on their response pattern across the six 
items, students are grouped into four engagement categories: none, limited, moderate, or high. Schools 
can examine percentage of students that fall into each category over time and compare their results with 
those of other schools. Schools outside Chicago Public Schools can model their reporting of results on 
the sample reports available on the CCSR website (Consortium on Chicago School Research 2007a, b,c,d). 


Reliability 


CCSR website reports Cronbach's alpha of .68 for elementary students and .65 for high school students 
on the AES. 


Validity 


Sebring et al. (1996) find that student academic engagement correlated positively with student-reported 
academic pressure (students' views of their teachers' efforts to push them to higher levels of academic 
performance) and perceived teacher social support (degree to which students feel their teachers offer 
personal support). As expected, high school students had lower mean scores on the AES than did 
elementary school students. CCSR reports studies under way to examine the correlations between the 
instrument and academic achievement measures. 


Use 


Most published information on use of the AES has been gathered by CCSR using samples from Chicago 
Public Schools (Consortium on Chicago School Research 2007a, 2007b, 2007c, 2007d). Data gathered 
during biennial student, teacher, and principal survey data collection are used primarily as monitoring 
tool. 



(CONTINUED) 
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TABLE A2 (CONTINUED) 

Academic Engagement Scale of the Consortium on Chicago School Research Biennial Survey (CCSR/AES) 



Feature Description 



References Consortium on Chicago School Research at the University of Chicago. (2007a). Sample Elementary School 

Detailed Report. Retrieved October 12, 2009, from http://ccsr.uchicago.edu/ISR/. 

Consortium on Chicago School Research at the University of Chicago. (2007b). Sample Elementary School 
Summary Report. Retrieved October 12, 2009, from http://ccsr.uchicago.edu/ISR/. 

Consortium on Chicago School Research at the University of Chicago. (2007c). Sample High School 
Detailed Report. Retrieved October 12, 2009, from http://ccsr.uchicago.edu/ISR/. 

Consortium on Chicago School Research at the University of Chicago. (2007d). Sample High School 
Summary Report. Retrieved October 12, 2009, from http://ccsr.uchicago.edu/ISR/. 

Sebring, P. B., Bryk, A. S., Roderick, M., Camburn, E., Luppescu, S., Thum, Y. M., Smith, B., and Kahne, J. 
(1996). Charting reform in Chicago: the students speak. Chicago: University of Chicago, Consortium on 
Chicago School Research. Retrieved October 12, 2009, from http://ccsr.uchicago.edu/publications/ 
ChartingReformlnChicago_TheStudentsSpeak.pdf 

Source: Authors' compilation. 
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TABLE A3 

Attitudes Towards Mathematics Survey (ATM): Cognitive Engagement in Academic Work Subscales 



| Feature 


Description 


Developer/website 


Developed and validated by Dr. Raymond B. Miller and colleagues at the University of Oklahoma. No 
website found. 


Availability/ 
key source 


Items for cognitive engagement subscales are shown in Miller et al. (1996), table 2. 


Population 


Administered to 297 suburban, southeastern high school students in their math courses. Versions of 
cognitive engagement items have also been used in studies with high school English students in a 
midwestern high school (Greene et al. 2004) and college-level samples (educational psychology students, 
preservice teachers, and students in statistics classes). 


Method 


Student self-report questionnaire. 


Background 


Developed by researchers studying relationship between various aspects of student motivation, 
engagement, and achievement. Original instrument developed and validated in 1993 (Miller, Behrens, 
and Greene 1993) and later revised by Miller et al. (1996). Cognitive engagement in academic work is 
measured by 28 of 83 items. Other variables measured included goals for doing academic work assigned 
in class and self-perceptions of ability for the class. Theoretical framework suggests that the type of 
cognitive strategies used by students in learning material correlates with achievement. Greene et al. 
(2004) reference "levels of processing" theory, which suggests that different strategies result in different 
achievement outcomes. Deep processing strategies are contrasted with shallow (rote) strategies. 


Administration 


Administered by: Teachers or members of research team. 

Form: Paper and pencil questionnaire. 

Time for 28 cognitive engagement items: About 15-20 minutes. 

Training/instructions: Because cognitive engagement items are worded to reflect strategy use in the 
particular content area, the instrument needs to be used after some instructional time has passed. 


Language 


English 


What is measured 


Contains five subscales assessing aspects of cognitive engagement in academic activities: 

• Self-regulation: 9 items (sample items: "Before a quiz or exam, 1 plan out how to study the material," 
"When 1 finish working a problem, 1 check my answer to see if it is reasonable"). 

• Deep cognitive strategy use: 7 items (sample items: "1 work several examples of the same type of 
problem when studying mathematics so 1 can understand the problems better," "1 classify problems 
into categories before 1 begin to work them"). 

• Shallow cognitive strategy use: 4 items (sample items: "1 find reviewing previously solved problems to 
be a good way to study for a test," "1 try to memorize the steps for solving problems presented in the 
text or class"). 

• Persistence: 8 items (sample items: "If 1 have trouble understanding a problem, 1 go over it again until 1 
understand it," "When 1 run into a difficult homework problem, 1 usually give up and go on to the next 
problem"). 

• Effort: 1 item with 5 alternatives (extremely high ["probably as much effort as I've ever put into a 
class"] to extremely low ["probably the least amount of effort I've ever put into a class"]). 


Scoring/reporting 


Users can create average scores for four subscales with multiple items, as items are grouped by subscale. 
Some items must be reverse scored. 


Reliability 


Cronbach's alpha scores (excluding the effort subscale, which had only one item) ranged from .63 to .81 
for the four subscales (Miller et al. 1996). 


Validity 


Miller et al. (1996) report that, of the four cognitive subscales, self-regulation had the strongest 
correlation with achievement; persistence was also significantly correlated with achievement. Deep 
cognitive strategy use was modestly correlated with achievement. Correlation between shallow cognitive 
strategy use and achievement was not significant. Miller et al. (1996) also report on construct validity 
(positive correlations of the cognitive engagement subscales with scales assessing students' perceived 
ability, beliefs about future consequences, and effort). 



Feature Description 
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TABLE A3 (CONTINUED) 

Attitudes Towards Mathematics Survey (ATM): Cognitive Engagement in Academic Work Subscales 



Feature Description 



Use Cognitive engagement subscales have been used primarily by educational psychologists conducting 

correlational studies examining the relationship among motivational variables, use of learning strategies, 
and achievement. Greene et al. (2004) adapt items for use with high school students in English classes. 
Greene and Miller (1996); Ravindran, Greene, and DeBacker (2005); and Miller et al.(1993) use cognitive 
engagement items in studies with college students. 

References Greene, B. A., and Miller, R. B. (1996). Influences on achievement: Goals, perceived ability, and cognitive 

engagement. Contemporary Educational Psychology, 21(2), 181-192. 

Greene, B. A., Miller, R. B., Crowson, H. M., Duke, B. L., and Akey, K. L. (2004). Predicting high school 
students' cognitive engagement and achievement: Contributions of classroom perceptions and 
motivation. Contemporary Educational Psychology, 29(4), 462-482. 

Miller, R. B., Behrens, J. T., and Greene, B. A. (1993). Goals and perceived ability: impact on student valuing, 
self-regulation, and persistence. Contemporary Educational Psychology, 78(1), 2-14. 

Miller, R. B., Greene, B. A., Montalvo, G. P., Ravindran, B., and Nichols, J. D. (1996). Engagement in 
academic work: the role of learning goals, future consequences, pleasing others, and perceived ability. 
Contemporary Educational Psychology, 27(4), 388-422. 

Ravindran, B., Greene, B. A., and DeBacker, T. K. (2005). Predicting preservice teachers' cognitive 
engagement with goals and epistemological beliefs. Journal of Educational Research, 98(4), 222-232. 

Source: Authors' compilation. 
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TABLE A4 

Engagement versus Disaffection with Learning (EvsD): student and teacher reports 



| Feature 


Description 


Developer/website 


Dr. Ellen Skinner, at Portland State University, provides access to most updated version of engagement 
scale (now called Engagement versus Disaffection with Learning scale) on her website (http://www.pdx. 
edu/psy/ellen-skinner) under "Assessments." Original engagement items were developed by Dr. James P. 
Connell and his colleagues at the University of Rochester (see Wellborn and Connell 1987; Wellborn 1991) 
as part of the Rochester Assessment Package. 


Availability/ 
key source 


Items are in appendixes A and B of Skinner, Kindermann, and Furrer (2009), available at http://www.pdx. 
edu/psy/ellen-skinner. 


Population 


Used with 1,018 students in grades 3-6 in suburban and rural schools. Items have also been used with 
samples of White and low-income racial/ethnic minority elementary, middle, and high school students in 
urban and suburban districts. 


Method 


Includes three methods: student self-report; teacher report on each student's engagement; and, for a 
subset of children, observations in the classroom. 


Background 


Student self-report and teacher report instruments were developed to assess components of a theory of 
student motivation (Connell 1990; Connell and Wellborn 1991; Deci and Ryan 1985; Skinner 1991; Skinner 
et al. 2009) that includes construct of engagement (versus disaffection) as key component of motivated 
action leading to student learning and achievement. Instruments assume that construct of engagement 
versus disaffection in the classroom ranges from enthusiastic, effortful, emotionally positive interactions 
with learning activities to apathetic withdrawal and frustrated alienation. Assessments include both 
positive manifestations of behavioral and emotional participation in classroom and withdrawal of 
behavioral and emotional participation and alienation from learning. Theoretical model of student 
motivation holds that student engagement (behavioral and emotional) is promoted when social context 
of learning meets students' basic psychological needs for relatedness, competence, and autonomy. 

Basic psychological needs are expected to be met in a context that provides involvement, structure, and 
autonomy support; neglectful, chaotic, or coercive learning contexts undermine students' needs and 
lead to disaffection. 


Administration 


Student report 

Administered by: Trained individuals read items to students. (In some studies, a second individual was 
present to monitor understanding and answer students' questions.) 

Form: Paper and pencil questionnaire, typically completed in classroom. 

Time: About 20-30 minutes. 

Teacher report 

Administered by: Researchers/evaluators provide copies of teacher report instrument (one for each 
student in their class) to participating teachers, who complete one rating form for each student. 

Time: A few minutes for each student in class. 


What is measured 


Student report contains 24 items in four subscales: 

• Behavioral engagement (5 items) taps effort, attention, and persistence in initiating and participating 
in learning activities (sample item: "When I'm in class, 1 listen very carefully."). 

• Behavioral disaffection (5 items) taps lack of effort and withdrawal from learning activities (sample 
item: "When I'm in class, 1 just act like I'm working."). 

• Emotional engagement (5 items) taps emotions indicating motivated involvement during learning 
activities (sample item: "1 enjoy learning new things in class."). 

• Emotional disaffection (9 items) taps emotions indicating withdrawal during learning (sample item: 
"When we work on something in class, 1 feel discouraged."). 

Teacher report contains items that are grouped in same subscales as student report, with each subscale 

containing four items: 

• Behavioral engagement (sample item: "In my class, this student does more than required."). 

• Behavioral disaffection (sample item: "In my class, this student comes unprepared."). 

• Emotional engagement (sample item: "In my class, this student appears happy."). 

• Emotional disaffection (sample item: "In my class, this student is anxious."). 



Feature Description 
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TABLE A4 (CONTINUED) 

Engagement versus Disaffection with Learning (EvsD): student and teacher reports 



Feature Description 



Scoring/reporting 


Subscale scores can be calculated as the average of items for subscale. For student self-report, response 
scale used ranges from 1 (not at all true) to 4 (very true). Structural analyses of item sets, conducted 
by Skinner, Kindermann, and Furrer (2009), indicate that items can be aggregated in different ways. 
Psychometric analyses show that subscales can be used separately or combined (by reverse coding the 
disaffection items and including them with the engagement items). Combinations include engagement 
(combining emotional and behavioral engagement items) versus disaffection (combining emotional and 
behavioral disaffection items) and behavioral (combining behavioral engagement and disaffection items) 
versus emotional (combining emotional engagement and disaffection items) engagement. Student 
profiles can also be reported (for example, high behavioral but low emotional engagement). 


Reliability 


Skinner, Kindermann, and Furrer (2009) report internal consistency reliabilities on a sample of students 
in grades 3-6 of .61-85 for the four student subscales over two administrations from fall to spring and 
reliabilities for four teacher report subscales of .81-87. Combining behavioral and emotional engagement 
items produced high levels of internal consistency (.79 and .86 for student report, .90 and .91 for 
teacher report). Interindividual stability was reported to be high. Cross-year correlations for subscales of 
engagement of .53-68 for student self-reports and .65-82 for teacher reports were reported. 


Validity 


Developers report evidence for construct validity through several methods. Skinner, Kindermann, and 
Furrer (2009) report that confirmatory factor analyses find that a four-factor model (distinguishing 
behavioral engagement, behavioral disaffection, emotional engagement, and emotional disaffection) 
was the best fit for both student and teacher report data. Four subscales correlated as expected: 
behavioral and emotional subscales correlated positively, and engagement and disaffection subscales 
correlated negatively. Teacher and student subscales showed modest agreement. (Agreement was higher 
for behavioral than emotional engagement.) Teacher reports correlated with external observer reports; 
student self-reports did not. Skinner et al. (2008) find expected age-grade patterns in which middle 
school students showed lower levels of engagement than did children in upper elementary school. 


Use 


Instruments have been used in published research that explores the multidimensional nature of 
engagement, changes overtime, predictors of engagement, and various student profiles of engagement 
(see references below). 


References 


Connell, J. P. (1990). Context, self, and action: a motivational analysis of self-system processes across the 
life span. In D. Cicchetti and M. Beeghly (Eds.), The self in transition. Chicago: University of Chicago Press. 

Connell, J. P., Halpern-Felsher, B. L., Clifford, E., Crichlow, W., and Usinger, P. (1995). Hanging in there: 
behavioral, psychological, and contextual factors affecting whether African American adolescents stay in 
school. Journal of Adolescent Research, 10, 41 -63. 

Connell, J. P., and Wellborn, J. G. (1991). Competence, autonomy, and relatedness: a motivational analysis 
of self-system processes. In M.R. Gunnar and L.A. Sroufe (Eds.), Minnesota Symposium on Child Psychology 
Voi. 23: Self-processes and development Chicago: University of Chicago Press. 

Deci, E. L., and Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. New York: 
Plenum Press. 

Skinner, E. A. (1991). Development and perceived control: a dynamic model of action in context. In 
M. Gunnar and L. A. Sroufe (Eds.), Minnesota symposium on child psychology Voi. 23: Self processes in 
development. Hillsdale, NJ: Erlbaum. 

Skinner, E. A., Kindermann, T. A., Connell, J. P., and Wellborn, J. G. (2009). Engagement as an organizational 
construct in the dynamics of motivational development. In K. Wentzel and A. Wigfield (Eds.), Handbook of 
motivation at school. Mahwah, NJ: Erlbaum. 

Skinner, E. A., Kindermann, T. A., and Furrer, C. J. (2009). A motivational perspective on engagement and 
disaffection: conceptualization and assessment of children's behavioral and emotional participation in 
academic activities in the classroom. Educational and Psychological Measurement, 69(3), 493-525. 

Skinner, E. A., Marchand, G., Furrer, C., and Kindermann, T. (2008). Engagement and disaffection in the 
classroom: part of a larger motivational dynamic. Journal of Educational Psychology, 100(4), 765-781. 

Wellborn, J. G. (1991). Engaged vs. disaffected action: conceptualization and measurement of motivation in 
the academic domain. Unpublished doctoral dissertation. Rochester, NY: University of Rochester. 

Wellborn, J. G., and Connell, J. P. (1987). Rochester Assessment Package for Schools, Student Report. 
Unpublished manuscript. Rochester, NY: University of Rochester. 



Note: See table A9 for a related item set. 
Source: Authors' compilation. 
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TABLE A5 

High School Survey of Student Engagement (HSSSE) 



| Feature 


Description 


Developer/website 


Developed by the Center for Evaluation and Education Policy (CEEP) at the Indiana University School of 
Education (http://www.indiana.edu/~ceep/hssse/). Dr. Ethan Yazzie-Mintz is the project director of the 
HSSSE. 


Availability/ 
key source 


The 2009 version is copyrighted. Upon registration, each school pays a nonrefundable participation/ 
user fee of $200 as well as a survey fee of $2 per student. Fees cover all survey materials, shipment of 
materials to the school, shipment of completed surveys back to the HSSSE office, scanning of completed 
surveys, and production and shipment of school data reports with survey results. A sample of the survey 
instrument can be found at http://www.indiana.edu/~ceep/hssse/survey.html. 


Population 


Piloted in 2003 with 7,200 students from four high schools, the survey has since been administered to 
large samples of high school students. In 2006 more than 100,000 students participated in the survey. 
From 2007 through 2009, the survey was administered to more than 200,000 students from rural, urban, 
and suburban school settings. 


Method 


Student self-report questionnaire. 


Background 


Since 2004 the HSSSE has been used to provide descriptive data on perceptions of national samples of 
high school students of their school work, the school learning environment, and their interaction with 
the school community. Periodic reports are available at http://www.indiana.edu/~ceep/hssse. The CEEP 
maintains a database of survey data from more than 300,000 students in more than 40 states. A middle 
school version of the instrument is being developed. 

The high school instrument builds on the National Survey of Student Engagement (NSSE), widely used 
to measure engagement and academic performance of college students. The NSSE has been used in 
developing national reports since the spring of 2000 (see http://nsse.iub.edu/index.cfm for additional 
information). 

Items on the 2008/09 HSSSE are described as assessing three dimensions of engagement: cognitive, 
intellectual, and academic engagement; social, behavioral, and participatory engagement; and 
emotional engagement. Survey can be used to guide planning for school improvement initiatives and to 
conduct long-term monitoring at the school or district level. 


Administration 


Administered by: School staff. 

Form: Paper and pencil questionnaire. 

Time: About 30 minutes. 

Training/instructions: Directions for students are included on the survey. 


Language 


English 


What is measured 


The 2009 version of the instrument includes 121 items that purport to measure three types of 

engagement: 

• Cognitive/intellectual/academic engagement (65 items) assesses students' perceptions of their effort 
toward, investment in, and strategies for learning. 

• Social/behavioral/participatory engagement (17 items) captures students' actions in social, 
extracurricular, and nonacademic school activities, including interactions with other students. 

• Emotional engagement (39 items) emphasizes students' feelings of connection (or lack of connection) 
to their school, including the ways and workings of the school and the people in their school. 


Scoring/reporting 


For schools purchasing HSSSE services, the CEEP provides analysis and reporting, including a customized 
report with mean comparisons and frequencies of student responses (item by item and by subscales) 
by grade, instructional track, race/ethnicity, and gender. A data file on a compact disc allows school 
personnel to conduct additional analyses. National HSSSE results and individual school results are 
reported for comparison, along with a guide for interpreting the mean comparison report. For schools 
seeking to improve student engagement, the CEEP offers technical assistance in selecting strategies for 
addressing identified issues. 


Reliability 


Studies conducted before 2008/09 generally presented survey results on an item-by-item basis and did 
not report reliability of internal consistency of subscales. Developers of the HSSSE considered reliability 
reported for the NSSE (Kuh 2004) as close approximation of the reliability of the HSSSE. As of fall of 2010, a 
reliability study of the HSSSE was in progress. 



Feature Description 
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TABLE A5 (CONTINUED) 

High School Survey of Student Engagement (HSSSE) 



Feature Description 



Validity 


As of the fall of 2010, a validity study of the HSSSE was in progress but not yet available. 


Use 


The HSSSE has been used in describing engagement at the high school level in CEEP annual reports and 
other published articles of national survey results. Schools and districts have also used the survey for 
school improvement planning (see, for example, Shafer 2008). National HSSSE findings have informed 
discussions on college readiness (McCarthy and Kuh 2006), dropout prevention (Azzam 2008), and 
graduation rates (Stanley and Plucker 2008). Several dissertations have used the HSSSE in examining 
aspects of high school functioning (Mayr 2008; Frazier 2008; Chang 2008; Lambert 2007). 


References 


Azzam, A. (2008). Engaged and on track. Educational Leadership, 65(6), 93-95. 

Chang, D. (2008). A case study to determine what perceived factors, including student engagement, 
contribute to academic achievement in a high performing urban high school. (Doctoral dissertation). 
Proquest Dissertations and Theses database. (Publication No. AAT 3311154). 

Frazier, K. (2008). Factors including student engagement impacting student achievement in a high 
performing urban high school district: a case study. (Doctoral dissertation). Proquest Dissertations and 
Theses database. (Publication No. AAT 3311148). 

Kuh, G. D. (2004). The National Survey of Student Engagement: conceptual framework and overview of 
psychometric properties. Bloomington, IN: Indiana University, Center for Postsecondary Research and 
Planning. 

Lambert, A. (2007). Student engagement in high-performing urban high schools: a case study. (Doctoral 
dissertation). Proquest Dissertations and Theses database. (Publication No. AAT 3278370). 

Mayr, K. (2008). A case study of factors related to a high performing urban charter high school: 
investigating student engagement and its impact on student achievement. (Doctoral dissertation). 
Proquest Dissertations and Theses database. (Publication No. AAT 3311156). 

McCarthy, M., and Kuh, G. D. (2006). Are students ready for college? What student engagement data say. 
Phi Delta Kappan, 87(9), 664-669. 

Shafer, J. (2008, November). KNIGHT line. Retrieved October 9, 2009, from http://www.menloschool.org/ 
data/files/gallery/KnightLineGallery/KnightLineNov2008.pdf. 

Stanley, K. R., and Plucker, J. A. (2008). Improving high school graduation rates. (Policy Brief 7). Bloomington, 
IN: Indiana University, Center for Evaluation and Education Policy. 

Yazzie-Mintz, E. (2007). Voices of students on engagement: a report on the 2006 high school survey of student 
engagement. Bloomington, IN: Indiana University, Center for Evaluation and Education Policy. Retrieved 
October 8. 2009, from http://www.indiana.edu/~ceep/hssse/images/HSSSE%200verview%20Report%20 
-%202006.pdf. 



Source: Authors' compilation. 
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TABLE A6 

Identification with School Questionnaire (ISQ) 



Feature Description 



| Feature 


Description 


Developer/website 


Developed by Dr. Kristin E. (Voelkl) Finn, School of Education and Human Services, Adolescence 
Education Department, Canisius College, Buffalo, NY. No website was found. 


Availability/ 
key source 


Information on instrument development, scoring, abbreviated items, and how items map onto constructs 
measured is available in Voelkl (1996). For the full instrument, see appendix B of Finn et al. (2007). 


Population 


Developed using a sample of 3,539 grade 8 students from 163 schools in rural, urban, suburban, and 
inner-city settings. About 25 percent of the sample was Black, and 75 percent was White. The instrument 
has since been used by other researchers with samples including American Indian, Asian, Black, and 
Hispanic students as well as low-income students in middle school and high school. 


Method 


Student self-report questionnaire. 


Background 


Developed from a theory of student engagement in school called the participation-identification model 
(Finn 1989, 1993), the instrument was created to examine relationships between school identification 
and other variables. Developers were particularly interested in developing an affective measure of 
emotional withdrawal from school that could predict later dropping out. Gender and race differences in 
disengagement were also of interest. 

The instrument consists of two subscales: belongingness with school and valuing of school. 
Belongingness is defined as the degree to which a student feels that he or she is a significant member 
of the school community. Valuing is defined as the importance a student places on learning and school. 
Although this instrument is a measure of identification with school, some researchers also consider it a 
measure of emotional engagement. 


Administration 


Administered by: Teachers or others. 
Form: Paper and pencil questionnaire. 
Time: About 15 minutes. 


Language 


Available only in English. 


What is measured 


Instrument comprises 16 items: 9 items on the belongingness with school subscale and 7 on the valuing of 
school subscale. Sample items for the belongingness subscale include "I feel proud of being a part of my 
school," "The only time 1 get attention in school is when 1 cause trouble," and "School is one of my favorite 
places to be." Sample items for the valuing subscale include "School is more important than most people 
think," "1 can get a good job even if my grades are bad," "Most of the things we learn in class are useless." 


Scoring/reporting 


A four-point response scale, ranging from 1 (strongly agree) to 4 (strongly disagree), is used for each item. 
Scale scores for individual students are obtained by reverse scoring necessary items, so that higher scores 
indicate a higher degree of identification with school, and then summing the scores of individual items. 
Students can be assigned a belongingness score or a valuing score (by summing the appropriate items) 
or an overall identification score (by summing all 16 items). 


Reliability 


Developer reports Cronbach's alphas of .76 for the belongingness scale and .73 for the valuing scale. 
Internal consistency for the identification measure as a whole is reported as .84 (Voelkl 1996). Other 
researchers have reported internal consistency reliabilities of .75-83 for the full scale, .66-78 for the 
belongingness scale, and .54-to .66 for the valuing scale (Bos et al. 2008; Kenny et al. 2003; Kenny et al. 
2006; Sirin and Rogers-Sirin 2005; Wettersen et al. 2005). 


Validity 


Confirmatory factor analysis by developer provides evidence for construct validity of two subscales 
(Voelkl 1996). Criterion-related validity has been demonstrated by positive associations of the 
identification measure with achievement and teacher ratings of classroom participation (Voelkl 1997). 


Use 


Voelkl (1996) examines gender and racial/ethnic differences in school identification. Other studies 
investigate the relationship between school identification and other variables, such as career 
development (Kenny et al. 2003; Kenny et al. 2006); performance and achievement within minority 
populations (Sirin and Rogers-Sirin 2005; Voelkl 1997), academic dishonesty and cheating (Voelkl-Finn 
and Frone 2004); substance use (Voelkl and Frone 2000); aggression at school (Finn and Frone 2003); 
and stereotype threat (Osborne and Walker 2006). Instrument was used initially as part of follow-up to 
the Project STAR longitudinal study of the effects of class size on achievement (Finn and Achilles 1990) 
to examine the long-term effects that reduced class size in early grades had on subsequent student 
identification with school. School districts have also used this measure to evaluate effectiveness of 
programs (such as the BioSMART magnet program evaluation in St. Paul Public Schools). 
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TABLE A6 (CONTINUED) 

Identification with School Questionnaire (ISQ) 



Feature Description 



References Bos, H. M. W., Sandfort, T. G. M., de Bruyn, E. H., and Hakvoort, E. M. (2008). Same sex attraction, social 

relationships, psychosocial functioning, and school performance in early adolescence. Developmental 
Psychology, 44(1), 59-68. 

Finn, J. D. (1989). Withdrawing from school. Review of Educational Research, 59, 117-142. 

Finn, J. D. (1993). School engagement and students at risk (NCES 93470). National Center for Education 
Statistics. Retrieved October, 1, 2009, from http://nces.ed.gov/pubs93/93470.pdf. 

Finn, J. D., and Achilles, C. M. (1990). Answers and questions about class size: a statewide experiment. 
American Educational Research Journal, 27, 557-577. 

Finn, J. D., Boyd-Zaharias, J., Fish, R. M., and Gerber, S. B. (2007). Project STAR and beyond: Database user's 
guide. HEROS, Inc. Retrieved October 1, 2009, from http://www.heros-inc.org/starllsersGuide.pdf. 

Finn, K. V., and Frone, M. R. (2003). Predictors of aggression at school: the effect of school-related alcohol 
use. NASSP Bulletin, 87(636), 38-54. 

Kenny, M. E., Blustein, D. L., Chaves, A., Grossman, J. M., and Gallagher, L. A. (2003). The role of perceived 
barriers and relational support in the educational and vocational lives of urban high school students. 
Journal of Counseling Psychology, 50(2), 142-155. 

Kenny, M. E., Blustein, D. L., Haase, R. F., Jackson, J., and Perry, J. C. (2006). Setting the stage: career 
development and the student engagement process Journal of Counseling Psychology, 53(2), 272-279. 

Osborne, J. W., and Walker, C. (2006). Stereotype threat, identification with academics, and withdrawal 
from school: why the most successful students of color might be most likely to withdraw. Educational 
Psychology, 26(4), 563-577. 

Perry, J. C. (2008). School engagement among urban youth of color: criterion pattern effects of vocational 
exploration and racial identity. Journal of Career Development, 34, 397-422. 

Sirin, S. R., and Rogers-Sirin, L. (2005). Components of school engagement among African American 
adolescents. Applied Development Science, 9(1), 5-13. 

Voelkl, K. E. (1996). Measuring students' identification with school. Educational and Psychological 
Measurement, 56(5), 760-770. 

Voelkl, K. E. (1997). Identification with school. American Journal of Education, 105(3), 294-318. 

Voelkl, K. E., and Frone, M. R. (2000). Predictors of substance use at school among high school students. 
Journal of Educational Psychology, 92(3), 583-592. 

Voelkl-Finn, K. E., and Frone, M. R. (2004). Academic performance and cheating: moderating role of school 
identification and self-efficacy Journal of Educational Research, 97(3), 115-122. 

Wettersen, K. B., Guilmino, A., Herrick, C. G., Hunter, P. J., Kim, G. Y., Jagow, D., Beecher, T., Faul, K., Baker, 
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TABLE A7 

Motivated Strategies for Learning Questionnaire (MSLQ), Cognitive Strategy Use and Self-Regulation 
Subscales 



| Feature 


Description 


Developer/website 


Developed by Dr. Paul Pintrich and Dr. Elisabeth DeGroot, School of Education at the University of 
Michigan. Initially developed for use with college students (Pintrich et al. 1991, 1993). Questionnaire 
adapted for use with middle schools by developer of college version (Pintrich and DeGroot 1990). No 
website was found. 


Availability/ 
key source 


Information on middle school version of the measure, including items and subscales, is available in 
Pintrich and DeGroot (1990). 


Population 


Validation of middle school version conducted using sample of 173 grade 7 students across 15 
classrooms. Middle school and college versions have been used in English- and non-English-speaking 
countries. In the United States, the instrument has been used by developers with primarily White middle- 
and working-class samples and by others with more diverse samples. 


Method 


Student self-report questionnaire. 


Background 


Initially designed as a self-report measure of college student learning as part of a grant to establish the 
National Center for Research to Improve Postsecondary Teaching and Learning, the college instrument 
contains 81 items assessing students' motivational orientation and use of cognitive and metacognitive 
strategies in a particular course. A 56-item version was adapted for middle school students. The middle 
school survey assesses motivational beliefs and self-regulated learning strategies. Cognitive strategy use 
scale and self-regulation scale have been used as measures of cognitive engagement in some studies. 
Measure designed to assess cognitive engagement in context of a particular class. 


Administration 


Administered by: Teachers or other appropriate adults. 

Form: Paper and pencil questionnaire. 

Time: 20-30 minutes. 

Training/instructions: Sample cover sheet with directions available in manual for college version (Pintrich 
etal. 1991). 


Languages 


Translated into Chinese, Farsi, Finnish, German, Greek, Hebrew, Korean, Norwegian, and Spanish. 


What is measured 


Middle school MSLQ contains five subscales, two of them relevant to student engagement. Cognitive 
strategy use has 13 items (sample items: "I outline the chapters in my book to help me study," "When 
1 study for a test, 1 practice saying the important facts over and over to myself"). Self-regulation has 9 
items (sample items: "1 ask myself questions to make sure 1 know the material that 1 have been studying," 
"Before 1 begin studying, 1 think about the things 1 will need to do to learn"). Subscales can be used 
together or individually. 


Scoring/reporting 


Seven-point scale ranges from 1 (not at all true of me) to 7 (very true of me). Scale scores created by 
averaging responses to all items in the scale, after reverse scoring negatively worded items. 


Reliability 


Research by developer and colleagues report Cronbach's alphas of .83-88 for cognitive strategy use scale 
and .63-74 for self-regulation scale (Pintrich and DeGroot 1990; Wolters, Pintrich, and Karabenick 2005). 
Other researchers report similar internal consistency reliabilities across range of samples. Two assessments 
measure reliability and validity in Chinese and Spanish samples (Duncan and McKeachie 2005). 


Validity 


Factor analyses conducted on data from middle school students yielded five factors (Pintrich and 
DeGroot 1990). Additional evidence of construct validity is reported from correlational studies conducted 
by developers and colleagues showing that self-efficacy, interest, and task value correlate positively with 
cognitive strategy use and self-regulation (Pintrich 1999). Criterion-related validity demonstrated through 
correlations of strategy use and self-regulation with indicators of academic performance, such as course 
assignments, exams, and grades (Pintrich and DeGroot 1990; Wolters and Pintrich 1998; Wolters, Yu, and 
Pintrich 1996). 


Use 


Uses of MSLQ are described in Duncan and McKeachie (2005). Middle school version has been used 
primarily in correlational studies examining relationship between motivation, use of learning strategies, 
and achievement (Pintrich and DeGroot 1990; Wolters and Pintrich 1998). Middle school version has also 
been used as outcome measure in examining impacts of different aspects of instructional strategies, 
course structures, and interventions (Duncan and McKeachie 2005). College version has been used 
widely by instructors to obtain feedback on students and by student learning centers as a form of needs 
assessment. 



Feature Description 
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TABLE A7 (CONTINUED) 

Motivated Strategies for Learning Questionnaire (MSLQ), Cognitive Strategy Use and Self-Regulation 
Subscales 



Feature Description 



| Feature 


Description 
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TABLE A8 

Motivation and Engagement Scale (MES) (formerly the Student Motivation Scale and the Student Motivation 
and Engagement Scale) 



| Feature 


Description 


Developer/website 


Developed by Dr. Andrew J. Martin, the University of Sydney, and published by the Lifelong Achievement 
Group (www.lifelongachievement.com). 


Availability/ 
key source 


Available from the Lifelong Achievement Group for about $85. Cost includes two-year license to 
administer the survey to maximum of 750 students, user manual, score sheets, norms for converting raw 
scores, data entry templates, and testing guidelines. Survey materials are provided by email as PDF and 
Microsoft Word attachments. Instrument is copyrighted by the Lifelong Achievement Group and may not 
be used without a license. 

Sample items and summary of administration, constructs measured, and scoring are available from 
Lifelong Achievement Group, at www.lifelongachievement.com/Motivation and Engagement Materials 
Summary 2009.pdf. 


Population 


The MES-Junior School (elementary/middle school) version has been normed in Australia with 1,249 
students, ages 9-13, across 63 classes in 15 schools. The MES-High School version has been normed with 
21,579 students, ages 12-18, across 58 schools. Samples were predominately middle-class students from 
urban, rural, and suburban areas of Australia. 


Method 


Student self-report questionnaire. 


Background 


Instrument assesses adaptive and maladaptive cognitive and behavioral dimensions of motivation and 
engagement. Summary here focuses on junior and high school versions only; separate versions are 
available for use with participants in university/college, workplace, sports, and music settings. 

Measure includes 11 subscales reflecting a multidimensional model of motivation and engagement. 
Developer integrated large body of research on student motivation and engagement into single 
framework of practical use to educators. Instrument is used as package for various purposes including 
assessment, benchmarking, tracking, research, and diagnosis to identify students' levels of engagement 
and disengagement. The 11 dimensions work together to create individual student profile. Student 
strengths and weaknesses are identified from pattern of results. Motivational/engagement weaknesses 
can then be addressed using developer's Motivation and Engagement Workbook (see www. 
lifelongachievement.com/workbook.htm; workbook requires an additional license fee), which includes 
weekly modules and exercises for students for targeted interventions for particular student profiles. 


Administration 


Administered by: Teachers, psychologists, counselors, researchers, or other professionals during a class 
period, in small groups or individually. 

Form: Paper and pencil questionnaire or web-based survey (with permission and appropriate security). 
For high school version, students complete survey on their own and return it at the end of class or 
session. For junior school (elementary/middle school) version, teachers read items to students one at a 
time as students complete them. 

Time: Up to full class period for the entire scale; 20-30 minutes if only certain subscales are administered. 

Training/instructions: No training is required to administer the instrument. Guidelines are included with 
the license. 


Language 


English 


What is measured 


Questionnaire contains 11 subscales, each of which comprises 4 items, for a total of 44 items. Sample high 
school items for each of the subscales include: 

• Self-belief: "If 1 try hard, 1 believe 1 can do my schoolwork well." 

• Learning focus: "1 feel very happy with myself when 1 really understand what I'm taught at school." 

• Valuing school: "Learning at school is important." 

• Persistence: "If 1 cannot understand my schoolwork, 1 keep trying until 1 do." 

• Planning: "Before 1 start a project, 1 plan out how I'm going to do it." 



Feature Description 
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TABLE A8 (CONTINUED) 

Motivation and Engagement Scale (MES) (formerly the Student Motivation Scale and the Student 
Motivation and Engagement Scale) 



Feature Description 



What is measured 
(continued) 


• Study management: "When 1 do homework, 1 usually do it where 1 can concentrate best" 

• Disengagement: "I've given up being interested in school." 

• Self-sabotage: "Sometimes 1 don't try hard at school so 1 can have a reason if 1 don't do well." 

• Anxiety: "When 1 have a project to do, 1 worry about it a lot." 

• Failure avoidance: "The main reason 1 try at school is because 1 don't want to disappoint my parents." 

• Uncertain control: "When 1 don't do well at school 1 don't know how to stop that happening next time." 


Scoring/reporting 


Response scale ranges from 1 (strongly disagree) to either 7 for high school version or 5 for middle school 
version (strongly agree). Responses to 4 items on each of the 11 subscales are aggregated as a raw score 
and then converted to a normed score (motivation quotient) for that dimension. Students are then 
assigned a grade from A to D on each construct, based on number of standard deviations below or above 
the mean score from the norming sample. Survey license comes with worksheets for calculating scores 
and developing profile for each student. 


Reliability 


Developer reports mean Cronbach's alpha for the 1 1 subscales as .79 for the high school version 
(.77— .82 for individual scales) and .78 for the junior school version (.70-87 for individual scales) (www. 
lifelongachievement.com; Martin 2009a, b). Average test-retest reliability for high school version is 
reported as .73 (.61 —.81 for individual scales) (Martin 2008b). 


Validity 


Developer has conducted confirmatory factor analyses to demonstrate construct validity of the 11 
subscales (Martin 2009a, b). Analyses demonstrate significant correlations with achievement and other 
academic outcomes, showing criterion-related validity. 


Use 


Developer has used instrument in various studies in Australia to evaluate effects of intervention programs 
(Martin 2005, 2007, 2008a), diagnose students with low motivation and engagement (Martin 2003), and 
examine differences between age and gender in motivation and engagement (Martin 2007). Howard 
(2006) conducted a study in the United States with a population of Black, urban high school students with 
low socioeconomic status, comparing the motivation and engagement of those promoted to grade 10 
with those retained in grade 9. 
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TABLE A9 

Research Assessment Package for Schools (RAPS) 



Feature Description 



Developer/website 


Adapted in 1998 by Dr. James P. Connell, cofounder and president of the Institute for Research and 
Reform in Education (IRRE; www.irre.org) from a longer instrument (called the Rochester Assessment 
Package for Schools) developed by Connell and Wellborn (1991) while at the University of Rochester. (For 
more recent version of the Rochester Assessment Package for Schools, see table A4 and the website of Dr. 
Ellen Skinner.) 


Availability/ 
key source 


Items are in the manual, which can be downloaded from www.irre.org at no cost. The IRRE offers fee-based, 
online administration of its surveys to cover costs of customization, coordination, and reporting of results. 


Population 


Student self-report questionnaire has been used with variety of urban and suburban adolescent 
populations. Evaluations of the First Things First school reform model included repeated administration 
of student self-report over time with elementary, middle, and high school students in Kansas City and 
four other suburban and urban districts (see www.irre.org). Items have been used with large populations 
of Black, Hispanic, White, and low-income youth in urban districts engaged in comprehensive school 
reform. Description here is of secondary school version. There is also an elementary version. 


Method 


Two methods: student self-report questionnaire and teacher report of individual student engagement. 


Background 


In 1998 the IRRE made RAPS available to reflect its overarching organizational approach to school reform, 
as articulated in its First Things First model. Assessment package focuses on student engagement as 
central construct to monitor in reform but includes other scales believed to reflect constructs that predict 
engagement. 

Instrument includes items assessing three constructs: student engagement, beliefs about self, and 
experiences of interpersonal support from parents and teachers. There is also a RAPS-T instrument, 
on which teachers report student engagement of each of their students. Developers note three main 
purposes: to provide diagnostic information on populations of students, evaluate school improvement 
interventions that seek to increase student engagement, and monitor changes in engagement and other 
important dimensions over time. 

The 1998 student self-report and teacher report on students RAPS are abbreviated from longer instruments 
developed by researchers led by Dr. Connell at the University of Rochester in the 1980s for use in research 
testing a theoretical model of engagement (Connell and Ryan 1984; Connell and Wellborn 1991). 


Administration 


Student self-report 

Administered by: Responsible adult other than student's teacher. 

Form: Paper and pencil questionnaire. 

Time: Middle school student instrument (84 items) takes about 50 minutes of class time to administer. 
Engagement subscale (11 items) takes significantly less time. 

Training/instructions: Manual strongly recommends training those administering the survey and reading 
aloud all items. 

Teacher report 

Administered by: Researchers/evaluators provide copies of the form to teachers. 

Form: Paper and pencil questionnaire. 

Time: About 1 minute per student. 

Training/instructions: Teacher should complete all forms (one for each student) at same sitting. 


Language 


English 


What is measured 


Student self-report 

Middle school student self-report measure has engagement scale with 11 items consisting of two 
subscales: ongoing engagement in school and reaction to challenge. Ongoing engagement in school 
includes five items that assess aspects of behavioral and emotional engagement. Reaction to challenge 
includes six items about strategies students use when faced with negative or stressful events. All 
items are on a four-point scale ranging from 1 (not at all true) to 4 (very true). Sample item for ongoing 
engagement: "I work hard on my schoolwork." Sample item for reaction to challenge: "When something 
bad happens to me in school [like not doing well on a test or not being able to answer an important 
question in class], 1 say it was the teacher's fault." 
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TABLE A9 (CONTINUED) 

Research Assessment Package for Schools (RAPS) 



Feature Description 



What is measured 
(continued) 


Teacher report on students 

Teachers complete three items on each student using a four-point scale ranging from 1 (not at all true) to 
4 (very true) of student: "In my class, this student seems tuned in," "This student comes to class prepared," 
and "This student does more than required " 


Scoring/reporting 


Ongoing engagement subscale score is average of responses to five items; reaction to challenge subscale 
score is average of six items. The 1 1 items can be combined into an overall engagement scale score. 

The teacher report measure results in a single score for each student that can then be averaged across 
students in a class. 


Reliability 


The manual reports internal consistency (Cronbach's alpha) for the engagement subscales of .68 for ongoing 
engagement, .69 for reaction to challenge, and .77 for the combined item set. Using 11 -item student self- 
report engagement subscales with a sample of Latino middle school students, Murray (2009) reports an 
internal consistency of .75. The manual reports internal consistency of .87 for the teacher report. Klem and 
Connell (2004) report similar internal consistency findings for both student self-report and teacher report. 


Validity 


The manual reports on criterion-related validity of student self-report measure and teacher report measure 
of engagement. Significant positive correlations are reported between engagement measures and student 
performance (for example, attendance, standardized test scores). Significant positive correlations reported 
between middle school students' scores on engagement scale and subsequent high school academic 
performance. Information on construct validity reported in the manual includes correlations between 
engagement and other variables (for example, engagement and student perceptions of teacher support); 
correlations between student and teacher reports of engagement; and factor analyses of the scale items. 


Use 


The IRRE reports that engagement scale has been administered to more than 200,000 students in 81 
elementary, middle, and high schools since 1997. Several evaluations of First Things First (implemented 
by the IRRE in urban districts) have been conducted by Manpower Development Corporation and Youth 
Development Strategies, Inc., using RAPS engagement items as outcome measure in assessing progress. 
Evaluations can be downloaded from the IRRE website (http://www.irre.Org/publications/#third_party). 
One research use of student engagement items (in study on parent and teacher relationships as 
predictors of student engagement in Latino youth) was identified (Murray 2009). 
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TABLE A10 

School Engagement Measure (SEM)-MacArthur Network 



| Feature 


Description 


Developer/website 


Developed by Dr. Phyllis Blumenfeld and Dr. Jennifer Fredricks, Connecticut College, as part of the 
MacArthur Network for Successful Pathways through Middle Childhood, (http://childhood.isr.umich.edu). 


Availability/ 
key source 


Items from three engagement subscales are available in Fredricks et al. (2005). 
Contact Dr. Fredricks (jfred@conncoll.edu) for permission to use measure. 


Population 


Used initially with urban, low-income, primarily Black and Hispanic sample of 641 students in grades 3-5 
attending neighborhood schools, the instrument has since been used in other studies of low-income, 
ethnically diverse upper elementary school students (Goldschmidt, 2008). 


Method 


Student self-report questionnaire. 


Background 


Developed for a study for the MacArthur Network for Successful Pathways through Middle Childhood, 
a multidisciplinary group of scholars studying students 6-12. Engagement items are part of larger 
longitudinal survey measure developed for study of relationship between classroom context and 
engagement among urban minority students in grades 3-5. Survey items include items drawn from 
variety of existing measures of motivation and classroom climate as well as new items. Measure has three 
engagement subscales (behavioral, emotional, and cognitive). 


Administration 


Administered by: Teachers or members of the research team. 

Form: Paper and pencil questionnaire. 

Time: School engagement items are part of larger survey that takes 30 minutes to administer. 
Training/instructions: Survey items should be read to students in class. 


Languages 


English and Spanish. 


What is measured 


Measure includes three subscales: five behavioral engagement items (sample items: "I pay attention in 
class," "I get in trouble at school" [reverse scored]); six emotional engagement items (sample items: "1 feel 
happy at school," "1 am interested in the work at school"); and eight cognitive engagement items (sample 
items: "When 1 read a book, 1 ask myself questions to make sure 1 understand what it is about," "1 read 
extra books to learn more about things we do in school"). 


Scoring/reporting 


Five-point scale ranges from 1 (never) to 5 (all of the time). Scale scores are created by averaging 
responses to all items in scale, after reverse scoring negatively worded items. 


Reliability 


Developers report Cronbach's alpha of .72-77 for behavioral engagement, .83-86 for emotional 
engagement, and .55-82 for cognitive engagement. In light of initial low reliability for cognitive 
engagement, developers made several changes in cognitive items from wave 1 to wave 2 (Blumenfeld 
et al. 2005; Fredricks et al. 2005). Other researchers report similar reliabilities for the three scales 
(Goldschmidt 2008). 


Validity 


Developers report several analyses that inform the construct validity of measure (Blumenfeld et al. 

2005; Fredricks et al. 2005). Factor analysis of items resulted in three subscales (behavioral engagement, 
emotional engagement, and cognitive engagement); the subscales correlated moderately with students' 
perceptions of aspects of their academic and social context, school value, and school attachment. 
Subscale scores were compared with data from interviews on engagement collected from the same 
sample of students. Developers report positive correlations between subscales and numerical ratings 
given to interview responses. 

Developers report expected subscale score differences by age (declines in scores with increasing age, 
from grades 3 to 5). Other researchers report that engagement subscales correlate positively with social 
skills and negatively with externalizing behaviors (Goldschmidt 2008). 


Use 


Used primarily in correlational research studies (Blumenfeld et al. 2005; Fredricks et al. 2005; Goldschmidt 
2008), items are also outlined in a Child Trends brief as a suggested measure of school engagement for 
out-of-school time practitioners. 



Feature Description 
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TABLE A10 (CONTINUED) 

School Engagement Measure (SEM)-MacArthur Network 



Feature Description 
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TABLE All 

School Engagement Scale (also known as School Engagement Questionnaire [SEQ]) 



| Feature 


Description 


Developer/website 


Developed by Dr. Sanford M. Dornbusch and Dr. Laurence D. Steinberg. Dr. Dornbusch is an emeritus 
professor in the Department of Sociology at Stanford University (http://www.stanford.edu/dept/soc/). 
Dr. Steinberg is a professor in the Department of Psychology at Temple University (www.temple.edu/ 
psychology/lds). 


Availability/ 
key source 


Perry (2008) and Kenny et al. (2003) describe the instrument. Copies of the questionnaire can be obtained 
from Dr. Dornbusch or Dr. Steinberg. 


Population 


Originally used with three-year longitudinal sample of about 12,000 students in nine high schools in 
Wisconsin and Northern California comprising a variety of socioeconomic statuses, ethnic backgrounds, 
and locales (Steinberg 1996), the instrument was subsequently used in studies with racially diverse high 
school students in rural and urban areas in the Northeast and Midwest. 


Method 


Student self-report questionnaire. 


Background 


Scale measures high school students' self-reported effort or investment in particular classes, as reflected 
in their time spent on homework assignments and their attendance, concentration, and attention in 
class. Original measure repeated same four items for each of three classes (math, English, social studies). 
Subsequent studies used eight items (four items repeated for just two classes, math and English). 

A large, national, descriptive study using the scale in nine high schools in Wisconsin and Northern 
California describes how parents, peers, and communities influence students' commitment to school 
(Steinberg 1996). The author uses several school engagement measures (including an emotional 
engagement work orientation scale) that may be of interest to researchers and educators. (This table 
describes only the school engagement scale, which was identified as being used in subsequent research 
and evaluations.) 


Administration 


Administered by: Teachers or other appropriate adults. 
Form: Paper and pencil questionnaire. 

Time: 10-20 minutes. 


Language 


English 


What is measured 


As used in studies reported here, the SEQ assesses a single construct (self-reported engagement in either 
two or three classes). The items (either 8 for 2 classes or 12 for 3 classes): 

• Homework: "How much time do you put into homework each week, including reading assignments?" 
(1 = None, 2 = About 15 minutes, 3 = About 30 minutes, 4 = About an hour, 5 = About 2 or 3 hours, 

6 = About 4 hours or more, 7 = Not taking this class) 

• Attendance: "How often do you cut (an unexcused absence) each of these classes?" (1 = Almost 
every day, 2 = Once or twice a week, 3 = A few times a month, 4 = A few times a year, 5 = Never cut, 

6 = Never taken this subject) 

• Classroom attention: "How often do you really pay attention during each of these classes?" (1 = Never, 
2 = Seldom, 3 = Fairly often, 4 = Usually, 5 = Always, 6 = Never taken this subject) 

• Classroom concentration: "How often does your mind wander in each of these classes?" (1 = Always, 

2 = Usually, 3 = Fairly often, 4 = Seldom, 5 = Never, 6 = Never taken this subject) 


Scoring/reporting 


Original scoring process used by developers involved first averaging scores for classroom attention and 
classroom concentration (which were assumed to measure same concept). The homework, attendance, 
and classroom attention/concentration scores were then summed to produce overall measure of 
engagement. Subsequent researchers have created composite scores by summing across all four 
questions to obtain an overall measure of engagement. In both cases, higher ratings indicate a stronger 
level of self-reported engagement with school classes. 


Reliability 


Original scale included four items for each of three subject areas (math, English, and social studies), with a 
reported Cronbach's alpha's of .86 (Dornbusch and Steinberg 1990; Taylor et al. 1994). Researchers using 
the eight-item version (for math and English classes) reported Cronbach's alphas of .74-80 (Perry 2008; 
Kenny et al. 2003). 
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TABLE All (CONTINUED) 

School Engagement Scale (also known as School Engagement Questionnaire [SEQ]) 



Feature Description 



Validity 


Criterion-related validity (in terms of positive correlations of the measure with grades) and evidence of 
construct validity (including correlations of engagement with student ratings of academic ability and 
perceptions of importance of school) were reported (Taylor et al. 1994). 


Use 


Four research studies use scale with high school students to understand relationships between variables 
such as self-reported vocational exploration, racial/ethnic identity, and engagement/identification with 
school (Perry 2008; Kenny et al. 2003. Taylor, et al. 1994; Wettersten et al. 2005). Instrument also used in 
program evaluation of Tools for Tomorrow in Massachusetts and included in evaluation plan as one of 
four measures to assess the level of school engagement for the students who participate. 
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TABLE A12 

School Success Profile (SSP) 



| Feature 


Description 


Developer/website 


Developed by Dr. Gary Bowen and Dr. Jack Richman, Jordan Institute for Families, School of Social 
Work, University of North Carolina at Chapel Hill, in partnership with Communities in Schools (www. 
schoolsuccessprofile.org or www.schoolsuccessonline.com/). 


Availability/ 
key source 


Instrument is available at www.schoolsuccessprofile.org or www.schoolsuccessonline.com/. It is 
copyrighted and may not be used without permission from developers. Costs for sites to use instrument 
depend on nature of request; charge to take online version is $2 per student, with a $350 registration fee 
per contract. (A single contract can include multiple schools or locations.) Sites are provided for access to 
online resource materials, administration, individual and group reports, and "evidence-based" practice 
strategies. The 2005 version of the instrument can be found in Bowen and Richman (2005). Additional 
information is available in Bowen, Rose, and Bowen (2005). 


Population 


Initially tested with 805 students at 26 middle schools in North Carolina (Richman, Rosenfeld, and Bowen 
1998), the instrument was subsequently used with more racially/ethnically diverse and low-income 
students in middle and high schools. 


Method 


Student self-report questionnaire. 


Background 


Survey is based on contextual perspective suggesting that social environment has powerful effect on 
child's development and success in school. Overarching purpose is to provide insight into how students 
perceive themselves and their environments, which can be useful for developing effective strategies for 
promoting student success. 

Initially developed in 1993, the survey has undergone five revisions (in 1993, 1997, 2001, 2005, and 
2008) to strengthen coverage of social environment domains and ensure manageable length for 
administration. It currently covers 22 core dimensions intended to assess students' perceptions of their 
social environment and their individual adaptation to it. Questions related to individual adaptation are 
divided into two categories: (a) personal beliefs and well-being and (b) school attitudes and behavior. 
School engagement and trouble avoidance subscales fall under the second category. 

Schools can use survey to identify areas in need of improvement, which can serve as impetus for 
implementation of interventions to address these areas. Survey can also assess how well intervention 
efforts are addressing areas of need. 


Administration 


Administered by: Trained member of school staff acting as onsite monitor. 

Form: Administered in school computer lab. 

Time: Depending on number of subscales used, survey may take up entire class period or longer. 

Training/instructions: Training materials for clients who register are available from developer's site. 
Administration and intervention training are available to clients through online webinars. Training 
requires online tool for real-time communication at remote sites. Onsite monitor enters unique serial 
number and password assigned to each student. Short video introduces students to survey and provides 
instructions for completing it. 


Languages 


English and Spanish; parts of the survey have been translated into Hebrew, Lithuanian, Portuguese, and 
Romanian 


What is measured 


School engagement subscale includes four school engagement items answered on a three-point scale 
ranging from "not like me" to "a lot like me." Sample items include "1 find school fun and exciting," "1 look 
forward to going to school," "1 look forward to learning new things at school," and "1 am often bored at 
school." Trouble-avoidance subscale assesses the extent to which youth report that they have avoided 
problem behaviors in the past 30 days that reflect getting into trouble at school. All 11 trouble avoidance 
items are answered on a "never in the past 30 days" to "more than twice in the past 30 days" three-point 
scale. Sample items include "1 turned in a homework assignment late or not at all" and "1 cut class at least 
once." 
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TABLE A12 (CONTINUED) 

School Success Profile (SSP) 



Feature Description 



Scoring/reporting 


All scoring is done by developer. School sites download summary profiles from the Internet. Access to 
profiles is password protected; sites have access only to their own data. Instrument provides data in the 
form of individual student profiles and site-level aggregate profiles. Integrated query system allows 
practitioners to examine group-level results within student subgroups. Each of the 22 dimensions 
on students or sites is assigned a color corresponding to cutoffs based on national norms, criterion 
analysis, and expert review. (The developer indicates 30 core dimensions; however, all publicly available 
information indicates 22.) Each dimension reflects continuum of protection from red (potential risk) to 
yellow (some caution may be warranted) to green (potential asset). Reference information on the school 
or child is included in profile result. 


Reliability 


Reports and articles indicate internal consistency of the two subscales, with Cronbach's alphas of .75-80 
for school engagement scale and .66-82 for trouble avoidance scale (Bowen, Rose, and Bowen 2005). 


Validity 


The developers and researchers report on the construct and criterion-related validity of the two 
subscales (Bowen and Bowen 1999; Bowen, Bowen, and Ware 2002; Bowen et al. 2008; Garcia-Reid 2007; 
Garcia-Reid, Reid, and Peterson 2005; Woolley and Bowen 2007). Student perceptions of teacher support, 
parent support, and peer support all correlate positively with student engagement. Neighborhood 
safety, for which high scores mean less safe neighborhoods, correlates negatively with student 
engagement. Trouble avoidance scores (high scores mean more troubled behavior) correlate positively 
with family poverty, negative neighborhood peer culture, students' personal experiences with crime 
and violence, and neighborhood social disorganization. Parental education support and supportive 
parenting correlate negatively with trouble avoidance. Correlations of the two subscales with academic 
achievement were not reported. 


Use 


Instrument has been used to provide needs assessment information to schools and districts to inform 
intervention and prevention planning (see www.schoolsuccessprofile.org). Website provides a handout 
on "strategies to strengthen school success profile dimensions," which offers ideas for how to address 
low scores on each of the 22 tapped dimensions. The survey has been used most extensively in North 
Carolina, where it was developed. 
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TABLE A13 

Student Engagement Instrument (SEI) 



| Feature 


Description 


Developer/website 


Developed by Dr. James Appleton, Gwinnett County Public Schools, Georgia, and Dr. Sandra Christenson, 
University of Minnesota. No website was found. 


Availability/ 
key source 


Scale development, validation, and items are available in Appleton et al. (2006) and Betts et al. 
(forthcoming). 


Population 


Scale development was conducted using a sample of 1,931 grade 9 students from an urban, ethnically 
diverse, majority low-income school district. Developer conducted later validation studies using students 
in grades 6-12. Other studies have also used instrument with middle school and high school students 
(Betts et al. forthcoming; Reschly et al. 2008). 


Method 


Student self-report questionnaire. 


Background 


Instrument consists of six subscales measuring two constructs: psychological engagement and 
cognitive engagement. As noted in program evaluations of the Check and Connect (www.ici.umn.edu/ 
checkandconnect) intervention developed to reduce dropout rates (Anderson et al. 2004), engagement 
is thought to consist of four subtypes: academic, behavioral, cognitive, and psychological. The 
Student Engagement Instrument was developed to go beyond observable indicators of academic and 
behavioral engagement (time on task, attendance, homework completion) to measure the cognitive and 
psychological aspects of engagement as reported by students. 


Administration 


Administered by: Teachers or other appropriate adults. 

Form: Paper and pencil questionnaire. 

Time: 20-30 minutes. 

Training/instructions: Developers suggest reading questions aloud to control for differences in reading 
ability. 


Language 


English 


What is measured 


Original questionnaire contained six subscales that measure psychological or cognitive engagement. 

Psychological engagement 

• Teacher-student relationships (9 items): Sample items: "Adults at my school listen to the student," "The 
school rules are fair." 

• Peer support for learning (6 items): Sample items: "1 have some friends at school," "Other students at 
school care about me." 

• Family support for learning (4 items): Sample items: " My family/guardian(s) are there for me when 1 
need them," "My family/guardian(s) want me to keep trying when things are tough at school." 

Cognitive engagement 

• Control and relevance of schoolwork (9 items): "The tests in my classes do a good job of measuring 
what I'm able to do," "Learning is fun because 1 get better at something." 

• Future aspirations and goals (5 items): "1 am hopeful about my future," "School is important for 
achieving my future goals." 

• Extrinsic motivation (2 items): "I'll learn, but only if the teacher gives me a reward." 


Scoring/reporting 


Four-point response scale ranges from strongly disagree to strongly agree. Negatively worded items 
are reverse scored. Scale scores are calculated by summing or averaging individual items. Item-scale 
mapping is described in Appleton et al. (2006). 


Reliability 


Appleton et al. (2006) report internal consistencies (Cronbach's alphas) of .88 for teacher-student 
relationships, .80 for control and relevance of schoolwork, .82 for peer support for learning, .78 for future 
aspirations and goals, .76 for family support for learning, and .72 for extrinsic motivation. Using a 25-item 
version of the instrument across five subscales, Reschly et al. (2008) find internal consistencies of .77-92. 



Feature Description 



(CONTINUED) 




APPENDIX A. INSTRUMENT ABSTRACTS 



47 



TABLE A13 (CONTINUED) 

Student Engagement Instrument (SEI) 



Feature Description 



Validity 


Appleton et al. (2006) demonstrate the construct validity of the six subscales using a confirmatory 
factor analysis. Betts et al. (forthcoming) confirm evidence of the validity of all subscales except extrinsic 
motivation. Engagement subscales correlate with measures of academic performance and behavior, 
demonstrating criterion-related validity through positive relationships with grade point average and 
reading and math achievement and negative relationships with frequency of suspensions. 


Use 


In addition to use by developer, instrument has been used in several research studies on engagement 
in school (Reschly et al. 2008; Lewis et al. 2009). Dr. Appleton has used instrument in Gwinnett County 
Public Schools in Georgia to create district, school, and individual student profiles to provide information 
to advisors that may supplement existing knowledge about students' achievement. 
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TABLE A14 

Student School Engagement Survey (SSES) 



| Feature 


Description 


Developer/website 


Developed by the National Center for School Engagement (NCSE), an initiative of the Partnership for 
Families and Children (www.schoolengagement.org). 


Availability/ 
key source 


Survey items and scale development information are available in National Center for School Engagement 
(2006a). 


Population 


Initially tested at three intervention sites with about 150 students (elementary school through high 
school), the instrument has been used with both ethnically diverse and low-income students. 


Method 


Student self-report questionnaire. 


Background 


Developed for use as outcome measure in evaluating interventions aimed at reducing truancy. The 
NCSE compiled items from pre-existing instruments, organizing them into three subscales: behavioral 
engagement, cognitive engagement, and emotional engagement. Survey focuses on psychological 
investment in learning, affective reactions in the classroom, and school conduct. 


Administration 


Administered by: Teachers or other appropriate adults. 
Form: Paper and pencil questionnaire. 

Time: 30-45 minutes. 

Training/instructions: No information available. 


Language 


English 


What is measured 


Questionnaire contains total of 45 items on three subscales. Items were adapted from several existing 
measures. Most items are answered on a "strongly agree" to "strongly disagree" scale. 

• Emotional engagement (16 items): Sample questions: "1 feel excited by the work in school," "Most of 
my teachers care about how I'm doing," "1 am happy to be at my school." 

• Cognitive engagement (22 items): Sample questions: "1 want to go to college, " "When 1 read a 
book, 1 ask myself questions to make sure 1 understand what it is about," "1 check my schoolwork for 
mistakes." 

• Behavioral engagement (7 items): Sample questions: "When 1 am in class, 1 just pretend 1 am working," 
"1 get in trouble at school," "1 try to stay home from school." 

Recent studies have used a shortened version of the questionnaire, with five engagement items per 
subscale. 


Scoring/reporting 


No information provided for scoring and reporting. However, items are mapped onto scales and can be 
summed to create scale scores. 


Reliability 


National Center for School Engagement (2006a) provides internal consistency measures of reliability for 
the three subscales, with Cronbach's alpha of .88-90 for emotional engagement, .87-92 for cognitive 
engagement, and .49-80 for behavioral engagement. The low reliability for the behavioral engagement 
scale was present at only one of the three intervention sites; values at the other two sites for this subscale 
were .79 and .80. A 2009 study conducted by the NCSE using five items per engagement subscale reports 
internal consistencies of .75-78 for behavioral engagement, .77-. 82 for cognitive engagement, and 
.81-83 for emotional engagement. 


Validity 


National Center for School Engagement (2006a) shows evidence of criterion-related validity of the 
subscales through positive correlations of the three subscales with grades and attendance. No factor 
analysis of items was provided, but three engagement subscales were reported to correlate positively. 


Use 


Reports on the NCSE website show use of the measure to evaluate the impact of truancy prevention 
initiatives on student engagement (Finlay and Heilbrunn 2006; NCSE 2006b) and to study the 
relationships between peer victimization, achievement, and attendance with student engagement as a 
mediating variable (National Center for School Engagement 2009). 
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TABLE A14 (CONTINUED) 

Student School Engagement Survey (SSES) 



Feature Description 
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Teacher report instrument 

TABLE A15 

Reading Engagement Index (REI) 



| | Feature 


Description 


Developer/website 


Developed by Dr. Allan Wigfield and Dr. John Guthrie, University of Maryland (www.cori.umd.edu). 


Availability/ 
key source 


Items are described in Wigfield et al. (2008). Items and format are available by contacting Dr. Wigfield 
(aw44@umail.umd.edu). 


Population 


Instrument was used with sample of 492 grade 4 students and with sample of 31 grade 4 students in mid- 
Atlantic schools . The student samples were roughly 60-70 percent White, 20-28 percent Black, and 5-7 
percent Hispanic. 


Method 


Teacher rating of each student's reading engagement. 


Background 


This teacher rating of the extent to which each a student is an engaged reader in the classroom 
was originally developed as an outcome measure for examining the effectiveness of a professional 
development intervention, Concept-Oriented Reading Instruction (CORI), designed by John Guthrie and 
colleagues to enhance students' reading engagement and increase reading comprehension (Guthrie, 
McRae, and Klauda 2007). Engagement was defined as having three dimensions (behavioral, emotional/ 
motivational, and cognitive). An engaged reader is assumed to be behaviorally active (reads frequently), 
internally motivated (likes to read), and cognitively active (uses strategies in reading). CORI teaches 
five instructional practices in reading that are "engagement supporting" to science or other content 
teachers (www.cori.umd.edu/what-is-cori/). It focuses primarily on upper elementary levels (grades 3-5). 
Guthrie etal. (2007a) summarize the results of 11 studies of the impact of CORI. In addition to reading 
comprehension scores, studies have examined the impact of CORI on other measures, including reading 
engagement of students as rated by teachers. 


Administration 


Administered by: Researchers/evaluators. 

Form: Paper and pencil questionnaire. 

Time: 20-30 minutes to rate all students in typical classroom. 

Training/instruction: Teachers should complete the ratings of their students in one sitting. 


Language 


English 


What is measured 


Instrument contains eight items intended to reflect the behavioral, motivational, and cognitive 
characteristics of engaged reading: 

• This student often reads independently. 

• This student reads favorite topics and authors. 

• This student is easily distracted in self-selected reading* (reverse scored). 

• This student works hard in reading* 

• This student is a confident reader* 

• This student uses comprehension strategies well .* 

• This student thinks deeply about the content of texts. 

• This student enjoys discussing books with peers. 

The authors have also used a shortened version of the instrument that includes the four items most 
correlated with students' self-reported motivation for reading as measured by the Wigfield and Guthrie 
(1997) Motivation for Reading Questionnaire (MRQ). These items are indicated above with an asterisk. 


Scoring/reporting 


Four-point response scale ranges from 1 (not true) to 4 (very true). Teacher's rating of each student's 
reading engagement on the items results in a total score of 8 to 32. If shortened four-item set is used, 
scores range from 4 to 1 6. 
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Reading Engagement Index (REI) 



Feature Description 



Reliability 


Wigfield et al. (2008) report internal consistency (Cronbach's alpha) reliability on a sample of grade 4 
students as .92; Cronbach's alpha for the four-item version of the scale was .89. Guthrie et al. (2007a) 
report reliabilities for a seven-item Reading Engagement Scale (REI) used in two prior studies of CORI 
impact as exceeding .93. 


Validity 


Wigfield et al. (2008) report that factor analysis indicates construct validity evidence supporting one 
factor. At individual student level, measure correlated positively with achievement as measured by both 
the Gates-MacGinitie measure and a text comprehension measure, thus establishing criterion-related 
validity. Guthrie et al. (2007b) show that teacher ratings of individual student reading engagement (on 
the REI) and student self-reports of motivation in reading (on the MRQ) are moderately correlated, a 
finding confirmed by Wigfield et al. (2008). 


Use 


Use of instrument has been cited in published research by the developers exploring the nature of reading 
engagement and motivation and the impact of the CORI intervention on reading engagement. 
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Observational instruments 

TABLE A16 

Behavioral Observation of Students in Schools (BOSS) 



j | Feature 


Description 


Developer/website 


Developed by Dr. Edward Shapiro, a school psychologist and the director of the Center for Promoting 
Research to Practice, Department of Education, Lehigh University (http://www.lehigh.edu/~ineduc/cprp/ 
news_asha_manz_grants.html). 


Availability/ 
key source 


The manual containing all the directions and forms needed to conduct the observations is in Shapiro 
(2004). Data can also be collected using a personal digital assistant available for purchase at www. 
pearsonassessments.com. The complete package can be purchased for $87.40. 


Population 


Developed for use with prekindergarten-grade 12 students, the instrument has been used in studies 
with ethnically diverse group of both typically developing and special needs populations. Most published 
studies have examined use with elementary school-age students. 


Method 


Systematic direct observation of individual students. 


Background 


Instrument measures observed individual student's on-task/off-task behavior or academic engagement 
time, using momentary/time sampling procedure continuously over 15-second intervals to record two 
categories of engagement and three categories of nonengagement. For example, a student observed 
for 15 minutes would have roughly sixty 15-second intervals of recorded data. Observer codes four of 
every five 15-second intervals. Multiple observations for each student are needed. Observed categories 
are summed across observations. Measure was developed for use by school psychologists for screening 
children, especially those at risk of academic failure, and for school psychologists, researchers, and 
evaluators tracking effectiveness of interventions over time. 


Administration 


Administered by: Trained observer 

Form: Paper and pencil version or version that uses a personal digital assistant, which enables observers 
to record observed categories electronically, allowing for easier analysis 

Time: Guidelines recommend a minimum of three observations of 20-30 minutes each, which should 
be repeated over 2-3 days. It is important to collect enough data points to obtain an accurate picture 
of student behavior. Guidelines indicate that number of observations needed depends on purpose 
and variability in behavior of the student. In some cases, observations may be needed across different 
academic settings (group work, seat work, and so forth). 

Training/instructions: Detailed guidelines for conducting the observations are available in Shapiro 
(2004). According to developer, training can be conducted through practice in classroom or with videos; 
interobserver agreement of 80 percent or better on all categories can be reached after training. 


Language 


English 


What is measured 


Each 15-second interval is coded along five categories of engaged time (two categories of engagement 
and three categories of nonengagement). Engagement categories include active engagement, 
coded when the observed student is actively engaged in assigned work (reading aloud), and passive 
engagement, coded when the observed student is passively attending to assigned work (listening to 
a lecture). Nonengagement is coded for an observed interval when any of the following is observed: 
off-task motor (engaging in any out of seat behavior), off-task verbal (making unauthorized comments 
and remarks), or off-task passive (staring out the window). At every fifth 15-second observed interval, 
a randomly selected comparison student in the classroom is observed. In addition to five categories of 
observed student behavior, there is a category for entering information on teacher-directed instruction. 


Scoring/reporting 


Scores from observations are percentages of occurrences of the five categories across total number of 
15-second intervals observed (percentage of intervals in which the category of engaged/nonengaged 
behaviors occurred). (See manual for full description of scoring.) 


Reliability 


In review of psychometric properties of the measure, Volpe et al. (2005) conclude that reports of 
interobserver reliability after training have been consistently high (90-100 percent). Using measure of 
academic engagement adapted from BOSS (on task/off task), Hintze and Mathews (2004) document high 
interobserver agreement (90 percent) but find that they could not achieve adequate reliability for some 
students until they observed them four times a day for four weeks. 



Feature Description 
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TABLE A16 (CONTINUED) 

Behavioral Observation of Students in Schools (BOSS) 



Feature Description 



Validity 


Volpe et al. (2005) report limited evidence of validity. There is some evidence that measure can 
differentiate between children with attention deficit hyperactivity disorder (ADHD) and typically 
developing children (DuPaul et al. 2004; Vile Junod et al. 2006); differences between the two groups were 
greater in observed off-task categories than in on-task categories. Spanjers, Burns, and Wagner (2008) 
report a small positive correlation between an adapted measure of BOSS and a self-report measure of 
engagement but suggest that more research is needed on the relationship. 


Use 


School psychologists have used instrument to supplement individual student assessment procedures 
(Hintz and Matthews 2004; Volpe et al. 2005). Researchers have used it to evaluate the effectiveness 
of education interventions on particular (targeted) students (Mautone, DuPaul, and Jitendra 2005; 
McQuillan and DuPaul 1996) and to identify differences between the academic behavior of typically 
developing children and children with ADHD (DuPaul et al. 2004; Vile Junod et al. 2006). 
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TABLE A17 

Classroom AIMS 



Feature Description 



| Feature 


Description 


Developer/website 


Developed by Alysia Roehrig while a graduate student at the University of Notre Dame. She is currently 
an assistant professor at the College of Education, Florida State University, (http://www.epls.fsu/edu/). 


Availability/ 
key source 


Copy of instrument can be obtained by contacting Dr. Roehrig at aroehrig@fsu.edu. 

For information on items, instrument development, reliability, and validity, see Roehrig and Christesen 
(forthcoming). 


Population 


Initially developed for use with elementary school teachers (K— 2) with varying levels of teaching 
experience, the measure has been used by the developer and her colleagues with both beginning and 
experienced public and private school elementary school teachers in primarily middle class schools. 
Instrument was used in one published study with secondary teachers in English, math, and science in 
an economically depressed, urban district (Stanulis and Floden, 2009). Developer has used it with both 
economically depressed public schools and middle class, rural, public elementary and secondary schools. 


Method 


Classroom observation. 


Background 


Measure was developed as a comprehensive approach to evaluating multiple domains associated 
with effective teaching practices. Items were identified from analyses of teachers who were successful 
in maintaining high levels of student engagement and literacy gains. Initial focus was on teaching 
dimensions associated with literacy engagement and achievement, but developers indicate that the 
items may be applicable across content areas. 

The four engagement items are part of the larger AIMS observation instrument designed to quantify 
teacher quality (by assessing teachers' use of practices consistent with those of exemplary teachers). It is 
not clear whether the engagement items can be used independently of the whole set of AIMS items. 


Administration 


Administered by: Experienced observers/evaluators. 

Form: Observation form. 

Time: Administering the full observation is time intensive. Developer initially recommended that two 
people observe the classroom and take extensive field notes over several visits (Roehrig 2003). Across 
studies using this measure, the number and time of observations ranges from two to five visits a year, for 
one to four hours each visit. The two observers meet to discuss observation and ratings, consulting their 
field notes to resolve disagreements until they agree on the completion of the items. More recently, in 
2010, developer indicates that observation can be conducted in 60-90 minutes (Roehrig and Christesen 
forthcoming). 

Training/instructions: No information on training observers was identified, although the developer 
indicates that training is important when using two observers. 


Language 


English 


What is measured 


Original version of instrument contained 130 items; most recent version includes 75, in three dimensions 
of classroom practices (atmosphere, instruction/content, management) and one dimension describing 
student engagement. Engagement is measured with four items, which focus on staying on task (at least 
80 percent of students are consistently on task and highly engaged in class activities; students are so self- 
regulated that disciplinary encounters are rarely seen); participating in class (students eagerly raise their 
hands and participate); and expressing excitement (students vocalize/express excitement about content/ 
activities — lots of oohs and aohs). 


Scoring/reporting 


Engagement items are rated using scale ranging from 1 (poor/rare use) to 3 (exemplar/consistent 
use). The four categories can be reported individually or together. Developer provides interpretative 
guidelines for overall results: teachers who score more than one standard deviation below the mean 
are characterized as "poor," teachers who score over one standard deviation above the mean are 
characterized as "exemplar," and teachers who score within one standard deviation of the mean are 
characterized as "typical" (Roehrig 2003). 


Reliability 


Across samples analyzed by developer and her colleagues, Cronbach's alpha's for the student 
engagement scale ranged from .62 to .79. Across 125 teachers observed by two people (9,350 unique 
observations), the developers (Roehrig and Christesen forthcoming) report exact agreement on 58 
percent; another 37 percent of ratings were off by only a point. In a study of secondary teachers, interrater 
reliability for individual items was 65 percent (Stanulis and Floden 2009), and internal consistency of 
scales was .60-95. (It was not clear which statistics correspond to the student engagement scale.) 



(CONTINUED) 




APPENDIX A. INSTRUMENT ABSTRACTS 



55 



TABLE A17 (CONTINUED) 

Classroom AIMS 



Feature Description 



Validity 


Initially, both academic experts and effective elementary school teachers examined the content validity 
of the categories, subcategories, and items and rated the importance of each item for exemplary 
teaching. Some items were dropped from the instrument as a result. Roehrig (2003) reports evidence 
of construct validity of the engagement scale through significant correlations with the three other 
categories of teacher practice. Confirmatory factor analyses by Roehrig and Christesen (forthcoming) 
provide support for the four overarching categories. 


Use 


Most published research has been conducted by developer and her colleagues. The measure has been 
used for a variety of purposes, including comparing more and less effective teachers (Bohn et al. 2004); 
studying the impact of mentoring programs on improving teacher quality in ways that link to student 
engagement (Roehrig 2003; Stanulis and Floden 2009); guiding professional development (Roehrig, 
Bohn, et al. 2008); examining the alignment of teachers' beliefs and practices (Roehrig et al. 2009); and 
guiding teacher reflection (Roehrig, Guidry, et al. 2008). 
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TABLE A18 

Code for Instructional Structure and Student Academic Response-Mainstream Version (MS-CISSAR) included 
in the Ecobehavioral Assessment Systems Software (EBASS) 



| Feature 


Description 


Developer/website 


Developed by Dr. Charles Greenwood and colleagues at the University of Kansas. Dr. Greenwood is 
the Director of the Juniper Gardens Children's Project at the University of Kansas (http://www.jgcp. 
ku.edu/~jgcp/products/EBASS/index.htm). 


Availability/ 
key source 


EBASS version 3.0 software, site licenses, manuals, and training videos are available for purchase at 
www.jgcp.ku.edu/~jgcp/products/EBASS/ebass_materials.htm.The EBASS software package, including 
two distribution disks, practitioner's manual, and technical manual, costs $350. A summary of the 
development and validation for the EBASS is found in Greenwood et al. (1994). A new EBASS mobile 
system and website are being designed. 


Population 


Developed and validated for use with both elementary and middle school students (Greenwood et al. 
1994), the instrument is relevant for observing students in both regular and special education classes. It 
has also been used to observe high school students (Wallace et al. 2002). 


Method 


Systematic direct observational measure of individual students in the classroom context. 


Background 


Instrument reflects program of research begun in 1981 to develop a better understanding of how student 
academic responding interacts with teacher behavior and classroom settings (Stanley and Greenwood 
1981, 1983). Original observational coding system (CISSAR) used 53 categories, defined in training manual 
and computer-assisted tutorial. The MS-CISSAR is an expanded version of the original observation 
categories, with event definitions organized into three categories. Student behavior (one of the three 
categories) focuses on the construct of engagement in terms of the observed student's "academic 
responding," considered a critical enabler of academic success and thus an important area for data 
collection (Greenwood, Horton, and Utley 2002). 

EBASS software was developed to make collection and analysis of observation data easier to input and 
analyze. Primary purpose of instrument is to collect classroom observation data on specific students for 
use by practitioners (for example, school psychologists) in improving instruction and results for students. 
It has also been used by researchers and program evaluators looking for an observational measure of 
student engagement. 


Administration 


Administered by: Trained observers working in the classroom using EBASS software. Momentary time 
sampling procedures used to prompt recording of observations. Onset of 20-second intervals is signaled 
by auditory bleep and event table on computer screen. Observers record events by typing their selection 
of the event names from the table (Greenwood et al. 1991). 

Time: Length of time of observation is driven by the need to adequately sample the situation in question. 

Training/instructions: Use of instrument requires training in applying the observation categories, offered 
by developer and through videotapes and other self-practice materials. 


Language 


English 


What is measured 


MS-CISSAR contains a 105-event taxonomy for recording observations. Taxonomy organizes events by 
student behavior, teacher behavior, and ecological setting. Under student behavior, observations are 
coded in terms of positive, neutral, or negative aspects of academic engagement, as follows: 

• Academic responding subcategory (positive engagement behaviors) is coded when student is 
observed engaged in work such as writing, playing an academic game, asking or answering an 
academic question, or reading (aloud or silently). 

• Task management subcategory (neutral engagement) is coded when student is observed in such 
behaviors as raising a hand to signal for help or looking for materials. 

• Inappropriate behavior/competing behavior subcategory includes undesired, inappropriate 
behaviors, such as being disruptive, talking inappropriately, or not paying attention. 


Scoring/reporting 


EBASS software program, which observers use to record their data, provides analytical tools for 
summarizing and displaying observation results. It can analyze data for individual students, individual 
observations, and pooled observations. Engagement analysis module shows plots across time (for 
example, student's observed academic responding percentage throughout the minutes of a school day). 



Feature Description 
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TABLE A18 (CONTINUED) 

Code for Instructional Structure and Student Academic Response-Mainstream Version (MS-CISSAR) included 
in the Ecobehavioral Assessment Systems Software (EBASS) 



Feature Description 



Reliability 


Greenwood et al. (1994) note that EBASS training materials were developed so that event definitions 
could be learned through drill and practice tutorials and that videotapes could be used for calibration 
(observers can evaluate percentage agreement against standards). Wallace et al. (2002) report observer 
agreement with the calibration video of 85-92 percent. In addition to evidence that observers can learn 
to calibrate against the standards, studies have reported interobserver agreements at acceptable levels 
after training (80 percent or higher in Rotholz, Kamps, and Greenwood [1989]; 80-90 percent in Kamps et 
al. [2008]; mean of 97 percent in Greenwood et al. [2002]). 


Validity 


Greenwood et al. (1994) and Greenwood et al. (2002) find evidence of criterion-related validity in that 
academic responding is positively correlated with academic achievement and mediates the effects of 
instruction on achievement. 


Use 


In addition to practical use of EBASS/MS-CISSAR by school psychologists to gather data on individual 
students' behaviors for clinical assessments. Greenwood et al. (2002) report that data on student 
engagement in academic responding have been used as outcome measure in studying alternative 
instructional approaches to reading, identifying classroom situations that promote engagement with 
students with disabilities, and examining engagement in high school classrooms recognized for inclusive 
practices. 
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TABLE A19 

Instructional Practices Inventory (IPI) 



Feature Description 



| Feature 


Description 


Developer/website 


Developed by Bryan Painter (1998) and Dr. Jerry Valentine, director of the Middle Level Leadership Center 
at the University of Missouri (http://www.education.missouri.edu/orgs/mllc). 


Availability/ 
key source 


Information about development and use of instrument (and future workshop locations for potential 
users) can be obtained by contacting Dr. Valentine (ValentineJ@missouri.edu). A new website devoted 
to disseminating information about the instrument (www. instructionalpracticesinventory.com) is under 
construction. 


Population 


First used in 1996 in eight elementary schools, eight middle schools, and eight high schools, the 
instrument was later used in preschools; vocational, technical, and career schools; and alternative schools 
in urban, suburban, and rural settings. 


Method 


Observation of classrooms (to be aggregated at the school level). 


Background 


Revised several times as new evidence and insights emerged (Quinn 2002; Valentine 2005; Valentine 
2007a), the instrument now uses a six-category rubric measuring the extent of student-engaged learning 
in a school. Process was developed as part of university-based school improvement project called 
Project ASSIST (described at www.mllc.org). Instrument was designed to produce a schoolwide picture of 
student-engaged learning that could serve as basis for faculty reflection and instructional change. Project 
ASSIST provided original context for development; instrument has since been used by hundreds of 
schools not associated with Project ASSIST. Intended for use by schools interested in gathering formative 
data about the extent of student-engaged learning as a basis for school improvement conversations (not 
for teacher evaluation or providing feedback to individual teachers). 


Administration 


Administered by: Trained data collectors in classrooms. 

Form: Observation. 

Time: Observations, usually taking three minutes, in all 100-120 classrooms in a typical school, are 
conducted in a single day. Schools may collect and discuss data three or four times a year. 

Training/instructions: Developer supports use of instrument only by people trained by certified 
trainers at one-day workshops, provided throughout the United States each year. IPI level 1 workshops 
focus on building capacity to collect valid, reliable data and capacity and to study the data collected. 
Recommended data collectors are teacher leaders in the school. As process is not linked to evaluation or 
supervision, school administrators are not viewed as appropriate data collectors. 


What is measured 


Observational categories were developed to describe broad range of engaged learning. "Student- 
engaged instruction" contains two coding categories that describe the type of learning students are 
experiencing (for example, project work, cooperative learning, small group discussions, and whole 
group discussions in which higher-order/deeper thinking is occurring among the majority of students). 
"Teacher-directed instruction" is coded when lectures, explanations by the teacher, worksheets, skill 
practice, or seat work are observed. "Disengagement" is coded as student or teacher disengagement. 


Scoring/reporting 


Observations are coded as one of the six types of teacher-student instructional engagement. Data 
analysis yields observation percentages for each of the six engagement levels. "Core" class and "noncore" 
class are profiled separately and together. Protocols for faculty discussion of the data are provided to 
enable reflection. Data from typical, exceptional, and struggling schools are available to give users a 
frame of reference against which to compare their schools' performance. 


Reliability 


Minimum rater reliability for the developer to endorse the use of the tool is .80 for site-based school 
improvement use and .90 for research use (Valentine 2007a). Developer offers workshops to train 
observers to reach the necessary.80 accuracy level. Observers who used only the website and other 
materials to learn the six coding categories had mean rater reliability scores of less than .20, compared with 
mean reliability scores of more than .90 by participants following workshop training (Valentine et al. 2006). 


Validity 


Valentine (2007b) finds significant differences between highly successful and unsuccessful middle schools: 
29.3 percent of observations in highly successful schools fell into the "student active engaged learning" 
category, for example, versus 16.0 percent in unsuccessful schools. Quinn (2002) reports that in a sample of 
24 schools teacher-rated principal leadership behaviors correlate positively with total IPI scores. 
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TABLE A19 (CONTINUED) 

Instructional Practices Inventory (IPI) 



Feature Description 



Use 


Primary use has been in school improvement processes. Developers report that 10,000 educators from 
hundreds of schools have been trained to use the IPI. Several recent dissertations (Gauen 2009; Collins 
2009) study implementation integrity and impact of use. Using a case study approach to explore the 
impact of the two-year use of IPI in an Illinois middle school, Gauen (2009) finds positive impacts. Using 
hierarchical linear modeling, Collins (2009) finds significant relationships between the integrity with 
which schools implemented the IPI collaborative processes and higher-order engagement and student 
achievement. Valentine and Collins (2009a, b) report that schools with high levels of implementation 
integrity and high levels of poverty have higher achievement gains than otherwise similar schools. 
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APPENDIX B 
METHODOLOGY 

This appendix describes the methodology used to 
search and screen for relevant instruments and 
then find and summarize information on the 21 
instruments that were identified. It explains how 
the instrument-documentation protocols were cre- 
ated and used and how the instrument abstracts 
that appear in appendix A were prepared. 



Executing the search 

To identify student engagement instruments, 
researchers first searched three databases: Aca- 
demic Search Premier, PsycINFO, and ERIC. All 
search combinations used the word “engagement.” 
The search was restricted to studies available in 
English and published between January 1979 and 
May 2009. The year 1979 was selected to predate 



the earliest emergence of engagement studies in 
the early 1980s. 

Database/keyword searches were completed and 
tracked in a set order, as shown in table Bl. For ex- 
ample, the first search was of the Academic Search 
Premier database, using “student engagement,” 
“school engagement,” and “measur*” as search 
terms. The second search was of the PsycINFO 
database and used the same keywords. The same 
procedure was then followed for ERIC. The search 
terms were then changed slightly (using “instru- 
ment” instead of “measur*”) and reported for the 
same three databases. 

The number of references identified per keyword 
search (each row in the table) is reported in the col- 
umn titled “total number of references identified.” 
The next column shows the number of unique 
references added by each search (row). The total 



TABLE Bl 

Database search results 



Search terms/database 


Number of 
references identified 


Number of unique 
references identified 


"student engagement" or "school engagement" and "survey" 


Academic Search Premier 


258 




183 


PsycINFO 


136 




65 


ERIC 


292 




144 


"student engagement" or "school engagement" and "assess*" 


Academic Search Premier 


144 




59 


PsycINFO 


188 




69 


ERIC 


286 




95 


"student engagement" or "school engagement" and "measur*" 


Academic Search Premier 


112 




112 


PsycINFO 


191 




163 


ERIC 


189 




146 


"academic engagement" and "measur*" or "instrument" or "questionnaire" 


or "survey" or "assess*" 


' or "scale" 




Academic Search Premier 


50 




29 


PsycINFO 


102 




54 


ERIC 


68 




27 


"student engagement" or "school engagement" and "questionnaire" 


Academic Search Premier 


18 




11 


PsycINFO 


61 




32 


ERIC 


63 




26 



(CONTINUED) 
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TABLE B1 (CONTINUED) 

Database search results 



Search terms/data base 


Number of Number of unique 

references identified references identified 


"student engagement" or "school engagement" and "scale" 


Academic Search Premier 


18 


3 


PsycINFO 


56 


9 


ERIC 


45 


6 


"student engagement" or "school engagement" and "instrument" 


Academic Search Premier 


17 


7 


PsycINFO 


41 


20 


ERIC 


46 


22 


"science engagement" and "measur*" or "instrument" or "questionnaire" 


or "survey" or "assess*" or "scale' 




Academic Search Premier, PsycINFO, and ERIC (searched concurrently) 


5 


2 


"math* engagement" and "measur*" or "instrument" or "questionnaire" or "survey" or "assess*" or "scale" 


Academic Search Premier, PsycINFO, and ERIC (searched concurrently) 


1 


1 


"language arts engagement" and "measur*" or "instrument" or "questionnaire" or "survey" or "assess*" or 


"scale" 


Academic Search Premier, PsycINFO, and ERIC (searched concurrently) 


1 


1 


"reading engagement" and "measur*" or "instrument" or "questionnaire' 


' or "survey" or "assess*" or "scale 


" 


Academic Search Premier, PsycINFO, and ERIC (searched concurrently) 


32 


28 


Total 


2,420 


1,314 



Note: Search engine used was EBSCOhost. 
Source: Authors. 



number of nonduplicated citations identified from 
the various search term combinations was 1,314. 



Screening citations and identifying 
potential instruments for inclusion 

A database of all 1,314 citations (including fields 
for author, title, and abstract) was created to track 
the citations during screening. The next steps in- 
volved screening the citations for potential instru- 
ments and reviewing the resulting instruments for 
inclusion/exclusion. 

Screening citations for potential instruments 

Piloting and training for the citation screening 
process. The project team initially developed a 
process for screening the 1,314 citations/abstracts 
using a sample set of 25 citations/abstracts. It 
jointly reviewed the 25 and developed possible 
coding categories to track the citation screening 



results. It then examined each citation/abstract to 
determine whether it was on or off topic and, if on 
topic, whether it mentioned an instrument. Some 
citations/abstracts contained a “named” measure 
or instrument, which could be identified easily; 
others referenced a measure but did not name 
it. The project team made a decision regarding 
obtaining the full article. In cases in which an 
instrument was mentioned but not named in the 
abstract, the project team obtained the full article 
to determine the status of the measure. The team 
reviewed a second set of 40 citations/abstracts to 
develop a common understanding of the decision 
rules in the citation screening process. A database 
format was developed to track the reviews of each 
citation/abstract. 

The project team then reviewed a third set of 25 
citations/abstracts. This time each team member 
coded the same citations/abstracts individually. 
Coding differences were then discussed, and a 
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common understanding of decision rules was 
agreed upon (for example, err in the direction of 
inclusion of the citation as on topic if there was 
any question). 

The last step involved each project team member 
individually reviewing and coding another set of 
20 citations/abstracts. For citations/abstracts on 
which coders were not in total agreement, group 
discussions of the reasoning led to consensus. 
Written directions for use of the database reflect- 
ing the agreed upon process were finalized. The 
database form included a reference number for 
each citation and four columns (not shown in 
table B2) taken directly from the search results 
(author, title, periodical, year). A column for inser- 
tion of the citation abstract was also included. 
Then, as shown in table B2, it also included col- 
umns for the citation screener to complete based 
on information in the abstract, or if needed, from 
a review of the retrieved full article. The content 
area (math, science, and so forth) was entered only 
for on-topic citations. 

Five project staff conducted the screening pro- 
cess for the 1,314 citations. These staff included 
one student of Dr. Fredricks, supervised by Dr. 
Fredricks, and four Regional Educational Labora- 
tory Southeast research staff members (two with 
Ph.D.s in educational research/evaluation and two 
with masters degrees in educational research). To 
ensure a common understanding, each citation 
screener first screened a new set of 50 citations and 
entered the information into the database. The 50 



database entries from each screener were reviewed 
with the principal investigators to address any dis- 
crepancies in completing the database field. This 
group check represented the final screening train- 
ing. The screeners then reviewed their portion of 
the citations, bringing questions to the principal 
investigators as they arose. 

After screening, a second staff member checked 
about half (578) of the screened citations for ac- 
curacy. That is, these coded citations/abstracts were 
checked by reassigning the sample of 578 to differ- 
ent screeners. The “checker” examined the database 
entries for each citation. If the checker was unable 
to agree with the original screener on the database 
entries on the citation, the citation was brought to 
the team for resolution. For the sampled citations 
checked, 548 (95 percent) had no discrepancies be- 
tween the screener and checker. The 30 discrepan- 
cies were discussed and resolved as shown below: 

• Citation coding changed from off topic to on 
topic: 12. 

• Citation coding changed from on topic to off 
topic: 13. 

• Named instrument identified that was not 
named in original coding: 2. 

• Discussion but coding not changed: 3. 

The main purpose in screening citations was to 
identify instruments mentioned or used. Of the 



TABLE B2 

Database form used to track instruments identified 
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TABLE B3 

Number and percentage of screened citations 




Source: Authors. 



1,314 citations identified from keyword searches, 

40 percent were coded as off topic; 36 percent 
contained a named instrument (table B3). Very 
few on-topic citations dealt with subject-specific 
engagement. Most citations coded as on topic dealt 
with general student or school engagement (88 
percent). 

The search/screening process yielded 144 unique 
instrument names (figure Bl). Twelve other 
instrument names were added to the list through 
informal supplemental processes (other contacts or 
work under way by the principal investigators on 
this topic, scans of references of various articles ex- 
amined, and so forth). As a result, 156 instruments 
were identified for possible inclusion in the study. 

Reviewing resulting instruments for inclusion/ 
exclusion. The project team reviewed the 156 
instruments for relevance. Criteria for exclusion 
were developed through team meetings during 
which relevant citations/abstracts (or full articles, if 
retrieved) on the 156 instruments were reviewed. 

The exclusion criteria excluded all but 19 instru- 
ment names in the next phase of the study. Subse- 
quent examination revealed that two of the named 
instruments (RAPS and the EvsD) included two 
measures of engagement (one a student self-report 
measure and one a teacher report on individual 



student engagement). Thus, the number of instru- 
ment abstracts in appendix A is 19, but the total 
number of instruments shown in figure Bl is 21 
(14 student self-report, 3 teacher report, and 4 
observational measures). 

Instruments were excluded for the following 
reasons: 

1. The instrument was developed for students 
outside the age range for this study (that is, 
preschool, kindergarten, early elementary, or 
college). 

2. The instrument was developed and used spe- 
cifically with special education populations. 

3. The instrument was developed before 1979, 
and little to no use of the instrument was 
documented between 1979 and 2009. 

4. The instrument measures a construct related 
to engagement (for example, school climate, 
school bonding, self-efficacy) but has not been 
referred to or used as a measure of engage- 
ment in subsequent studies (table B4). 

5. The instrument was used as part of a large- 
scale survey in which there were only a few 
engagement items— not used as a scale— and 
engagement was not the main focus of the 
study (table B5). Eleven large-scale surveys 
were excluded because the engagement-related 
items were difficult to access. This occurred 
for various reasons: items are scattered 
throughout the larger survey instrument 

and not identified as an engagement scale; 
subsequent researchers analyzed the survey 
data in secondary analyses by combining 
engagement items from the survey in different 
ways; and there is limited descriptive infor- 
mation of the engagement items apart from 
the larger survey. Seven survey instruments 
contain a parent report and focus on family 
topics. Examining these large-scale studies 
may yield useful engagement items scattered 
throughout the instruments, but the items are 
not grouped into an engagement scale. 
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FIGURE B1 

Processes for screening citations and reviewing instruments for inclusion/exclusion 




a. Two of the 19 instrument profiles included in appendix A contain both a student self-report and a teacher rating component They are counted in both 
categories here. 



Source: Authors. 
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TABLE B4 

Instruments excluded because focus is not on engagement 



Construct measured 


Instrument 


References identified in search 


Attitudes toward school 


Positive Orientation to School 


Jessor et al. (1995) 


School Attitude Measures (SAM) 


Dolan (1983) 


Burnout 


School Burnout Inventory (SBI) 


Salmela-Aro et al. (2009) 


Classroom as community 


Rovai Classroom Community Scale (CCS) 


Lear (2008) 




Classroom Environment Scale 


Tobin and Fraser (1990) 


Classroom/learning environment 


System for Teaching and Learning 
Assessment and Review (STAR) 


Claudet and Ellett (1990); Heroman (1990) 


Connection to school 


School Connection 


Brown and Evans (2002) 


Effectiveness of instruction 


Diagnostic Classroom Observation (DCO) 


Saginor (2008) 


Flow 


Flow Activities Assessment 


Whalen (1997) 


Flow State Scale 


Fullagar and Mills (2008) 


Goal orientation 


Goal Orientation and Learning Strategies 
Survey (GOALS-S) 


Dowson and Mclnerney (1997) 


Patterns of Adaptive Learning Scale (PALS) 


Karam (2006); Lippman and Rivers (2008); 
Maehr and Midgley (1996) 




Locus of control, attendance 


The Academic Ethic 


Pino and Smith (2004) 


Metacognition 


Metacognitive Questionnaire (MQ) 


Howard-Rose and Winne (1993) 


Motivation for reading 


Motivations for Reading Questionnaire 
(MRQ) 


Wigfield and Guthrie (1997) 


Parent involvement 


Perception of Parent Involvement 


Abd-EI-Fattah (2006) 




School Practices and Policies Assessment 


Christenson (1995) 


Policies related to dropping out 


Worksheet and Family Practices and 
Policies Assessment Worksheet 




Rating of behavioral problems 


Teacher-Child Rating Scale 


Luthar and Ansary (2005) 




School Bonding 


Jenkins (1997) 


School bonding 


School Bonding 


Hawkins, Battin-Pearson, and Abbott (2001) 




School Bonding Questionnaire 


Hudson (1999) 


School climate 


School Climate 


Simons-Morton and Crump (2002) 


School Climate Questionnaire 


Blackadarand Nachtigal (1986) 


School membership 


Psychological Sense of School Membership 
(PSSM) 


Goodenow (1992); Shochet et al. (2006); 
Shochet et al. (2008); Voelkl (1996) 


Self-concept 


Children's Self-Concept Scale 


Hodges and Wolf (1997) 


Self-efficacy 


Science Self-Efficacy Questionnaire (SSEQ) 


Miller (2006) 


Self-regulated learning 


Self-Regulated Learning Rating Scale (SRL) 


Howard-Rose and Winne (1993) 


Social competence 


Social Competence Scale 


Hughes and Zhang (2007); Luo et al. (2009) 


Social Skills Rating System 


Sinclair etal. (1998) 


Strategy use 


Learning Process Questionnaire 


Ainley (1993) 


Time perspective 


Time Perspective 


Horstmanshof and Zimitat (2007) 



Source: Authors' compilation. 
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TABLE B5 

Instruments excluded because part of a large-scale survey 



Instrument/study 


Population 


Method 


Website 


California Healthy Kids 
Survey 


Students in grades 5-12 in 
California 


Paper and pencil student self-report 
questionnaire 


www.californiahealthykids.org 

www.wested.org/hks/ 


Educational Longitudinal 
Study (ELS) 


Students in grade 10 
through early adulthood 


Paper and pencil student self-report 
questionnaire; teacher and parent 
report questionnaires; 
follow-up student self-report 
conducted by web, telephone 
interview, or computer-assisted 
personal interview 


nces.ed.gov/surveys/ 

ELS2002/ 


Longitudinal Surveys of 
Australian Youth (LSAY) 


Students age 15 through 
early adulthood in 
Australia 


Student self-report conducted by 
telephone interview 


www.lsay.edu.au/ 


Minnesota Twin Family 
Study (MTFS) 


Twins age 11 through 
adulthood and their 
parents 


Face-to-face interviews with twins 
and parents; physiological exams 


mctfr.psych.umn.edu/ 

twinstudy/ 


National Education 
Longitudinal Study (NELS) 


Students in grade 8 
through early adulthood 


Paper and pencil student self-report 
questionnaire; teacher and parent 
report questionnaires; follow-up 
student self-report conducted by 
telephone interview or computer- 
assisted personal interview 


nces.ed.gov/surveys/NELS88/ 


National Longitudinal 
Survey of Children and 
Youth (NLSCY) 


Canadians from birth 
through early adulthood 


Parent reports conducted by 
computer-assisted personal 
interview (ages 0-17); student self- 
reports conducted by computer- 
assisted personal interview (ages 16 
and older); paper and pencil student 
self-report questionnaire (ages 
12-17); cognitive tests 


www.statcan.gc.ca/imdb- 

bmdi/4450-eng.htm 


National Survey of 
America's Families (NSAF) 


Families across 13 states 


Parent reports conducted by in- 
person interviews and telephone 
interviews 


urban.org/center/anf/nsaf.cfm 


National Survey of Child 
and Adolescent Well- 
Being (NSCAW) 


Families that enter child 
welfare system with 
children ages 0-14 


Teacher report questionnaires 


www.acf.hhs.gov/programs/ 

opre/abuse_neglect/nscaw/ 


National Survey of 
Children's Health (NSCH) 


Households with children 
ages 0-17 


Parent reports conducted by 
telephone interview 


www.nschdata.org 


Philadelphia Educational 
Longitudinal Study (PELS) 


Students in grade 8 
through early adulthood 


Student self-reports and parent 
reports conducted by telephone 
interview 


www.philaedfund.org/ 

research/research-studies/ 

philadelphia-educational- 

longitudinal-study-pels 


Third International 
Mathematics and Science 
Study (TIMSS) 


Students in grades 4-8 


Paper and pencil student self- 
report questionnaire; teacher and 
administrator report questionnaires; 
mathematics and science 
assessments 


http://nces.ed.gov/timss/ 



Source: Authors' compilation. 
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6. There was not enough information on the 
instrument to complete an instrument 
abstract (only one reference was cited, insuf- 
ficient descriptive information was provided, 
items were difficult to find; table B6). Twenty 
instruments were excluded because of 
inadequate information, 17 of them because 
they were cited only once in the original 
search. Of these 17, a follow-up web search for 
additional information was conducted for in- 
struments whose names suggested that they 
would be relevant for inclusion. The follow-up 



search was performed using the name of the 
instrument and author cited before conclud- 
ing that there were no additional citations 
other than those listed. For the three instru- 
ments in table B6 with multiple references, 
explanations for their exclusion are provided 
in the table notes. 

7. Measure was developed for use in non- 
academic subject areas (such as physical 
education) or adapted from an instrument 
already included (duplication). 



TABLE B6 

Instruments excluded because of inadequate information 



Instrument excluded/reason for exclusion 


References identified in search 


Inadequate information 


Academic Engaged Time (AET) a 


Walker and others (1993) 


Adolescent Family and Social Life Questionnaire 


Lin and others (2008) 


Association of Mothers' Report of Family Management 
Practices 


Taylor and Lopez (2005) 


Check and Connect Monitoring Sheet 


Anderson and others (2004) 


Engagement Check II (adapted from Planned Activity 
Check- PLAC) a 


Ridley and McWilliam (2001) 


Engagement in School — Teacher Rating Scale 3 


Anderson and others (2004) 


Engaging Behavior Checklist 3 


Beuscher and others (1997) 


KIDS COUNT report 


Farber and Burgess (2002) 


School and Social Experiences Questionnaire (SSEQ) Subscale 


Singh, Chang, and Dika (2007) 


School Transition Study (STS) 


Hauser-Cram and others (2006) 


Secondary Student Opinion Survey 


Sinclair and others (1998) 


Student Engagement in the Mathematics Classroom Scale 3 


Kong, Wong, and Lam (2003) 


Student Engagement Questionnaire 3 


Marks (2000) 


Student Engagement Rating Instrument-Anderson Method 3 


Kumar (1991) 


Tell Them From Me Survey 


Willms and Flanagan (2007) 


Transaction in Science (TIS) 


Kumar (1991) 


Youth Risk Behavior Survey 


Carter, McGee, Taylor, and Williams (2007) 


Other reason 


Perceived Academic Engagement Scale (PAES) b 


Chen (2005, 2008); Jeon (2008) 


Student Engagement and Family Culture Survey 0 


Leithwood and Jantzi (1998, 1999a, 1999b, 1999c, 2000) 


Utrecht Work Engagement Scale for Students (UWES-S) d 


Gan et al. (2007); Kiuru et al. (2009); Schaufeli et al. (2002); Wefald 
and Downey (2009); Zhang, Gan, and Cham (2007) 



a. A follow-up web search for additional information was conducted because the instrument's name suggested that it would be relevant for inclusion. None 
was found. 

b. Instrument appears to have been validated only in a language other than English. 

c. Multiple iterations of scale items were difficult to describe. 

d. Original scale was developed for use in the workplace and revised for students; information describing use with targeted age group was not available. 
Source: Authors' compilation. 
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Organizing citations by instrument and 
searching for additional information 

For each of the 19 instrument names identified 
for inclusion, a reference list was created using the 
relevant citations identified in the search process. 
The next step involved collecting enough addi- 
tional information on each instrument to describe 
it. The project team developed a systematic process 
for gathering basic information about each instru- 
ment. The goal was not to conduct an exhaustive 
search on each instrument but to find key manuals 
or other articles describing the development, fea- 
tures, and psychometric properties of the measure 
and to gather enough information to develop an 
instrument abstract. 

The searches were conducted by the five mem- 
bers of the research team that screened the 
citations using a common checklist. This search 
used EBSCOhost (an academic search engine) to 
search three databases (Academic Search Premier, 
PsycINFO, and ERIC) on each instrument name, 
instrument abbreviation, and author name. The 
citations were reviewed by the research team mem- 
ber conducting the search who made sure that rel- 
evant references not already identified were added 
to the reference list for the instrument. The team 
member also completed a web search to identify 
key websites for each instrument. As a result of the 
additional searching, a folder for each instrument 
was compiled including the following data: 

• Instrument fact sheet (name, developer, con- 
tact information, website). 

• Copy of instrument. 

• Copy of instrument manual or key article 
describing its development. 

• Copies of articles on reference list generated. 

• Copies of web-based information specific to 
the instrument. 



Completing the instrument-documentation 
protocols and creating the instrument abstracts 

The principal investigators and project team created 
the instrument-documentation protocol to pull 
common categories of information on each instru- 
ment from the various sources in the folders and 
share the results with the team. To learn how to 
complete the protocol, the project team completed 
instrument-documentation protocols on three sepa- 
rate instruments. Working through the completion 
of these three protocols together with the principal 
investigators led to the development of the direc- 
tions and definitions shown in table B7. Project 
team members then completed the instrument-doc- 
umentation protocols on two instruments assigned 
to them as a practice run. Together the principal 
investigators and team reviewed the completed 
protocol and provided feedback. The remaining pro- 
tocols were then assigned to five team members. 

Completed protocols were used to create shorter 
instrument abstracts. The final set of categories 
used in the instrument abstracts included in 
appendix A evolved over several team meetings, 
during which the team compared first drafts by 
team members and developed a common under- 
standing of what information belonged in each 
category (row) of the abstracts. Three levels of 
review were conducted to ensure accuracy across 
the 19 abstracts: 

1. Each abstract was reviewed by a second 
team member against the materials in the 
instrument folder, including the completed 
instrument-documentation protocol and 
copies of articles, noting any questions or in- 
consistencies, which were then used to revise 
the description. 

2. The entire set of abstracts was reviewed by the 
two principal investigators (with the instru- 
ment folders of materials available) for accu- 
racy, clarity, common language, and common 
levels of information. 



The instrument reference lists are included in the 3. After the principal investigators reviewed and 

references of this report. finalized the abstracts, the abstracts were sent 
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to the developers for review. The developers 
were informed of the purpose of the study and 
asked to review the abstract for accuracy. Of 
the 19 abstracts in appendix A, 18 were re- 
viewed by the developer. Four developers made 
no changes to the information; 14 provided 
additional references or updates to the instru- 
ment information. The developers of the MSLQ 
were contacted twice but did not respond. 



Once the instrument abstracts were complete, 
analysis involved summarizing the numbers and 
kinds of instruments identified and examining 
information useful to practitioners and evaluators/ 
researchers. The principal investigators worked 
together to review information provided in all 
instrument abstracts and draff the findings. 



TABLE B7 

Instrument-documentation protocol 



Is the instrument manual available? 
□ Yes □ No 



Information domain 


Directions 


1 . 


Instrument name 


Provide the actual instrument name. 


2. 


Instrument description 


Provide a brief, several sentence overview description of the instrument named 
above. For example, number of items, all subscales (indicate which subscale we 
are interested in if engagement is only a piece of larger instrument), and format of 
instrument (for example, student self-report, teacher observation) 

The description should answer the question — what is it? 

Cite the source for the descriptive information. 


3. 


Publisher/developer and website (if 
applicable) or citation 


Provide publisher, developer, or author contact information. Include information on 
website or other address for obtaining or purchasing the measure. If measure is only 
available in a published article, cite the article. 


4. 


Development context/history 


This should be a summary of what is known about the development context/ 
research history for the instrument. 

For example, was it developed as part of a research project or was it developed 
for use in improving teacher quality, or to provide more insight into student 
perspectives on school? 

Provide the timeline for the instrument as you can infer it from various sources 
(initially developed in. . .., revised in. . ., changed from . . . to . . . over time, and so 
forth). If it was initially developed for research applications and then moved into 
school reform/practical applications or if it was initially developed at the college 
level and then moved into K-12, those kinds of historical contexts should be noted. 

If there is a particular author/researcher that has been central to the development 
of the instrument, note that influence/involvement. If it emerged out of a particular 
research line/discipline (educational psychologists working on locus of control, for 
example), note that. 


5. 


Theoretical framework 


If a particular theoretical framework is cited, briefly describe that, and the citation 
where the theoretical framework is discussed. 


6. 


Definition of engagement 


This is a place to put quotes/notes about the definition of engagement that the 
instrument seems to be based on and whether behavioral, emotional, cognitive 
engagement, or other similar terms are mentioned. Cite the source of the quotes or 
definition of engagement. 


7. 


Stated/intended purpose for 
engagement measure 


This should be information that comes directly from the manual or seminal article/ 
description of the development of the measure. 



(CONTINUED) 
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Instrument-documentation protocol 



Information domain Directions 



1 Information domain 


Directions 


8. Populations of focus 

Intended grade levels (quote 
source) 

List populations for which use has 
been reported 

Norming or comparison 
information available 


Describe here anything that is known about the populations of focus: 

For individual articles, cite the grade range used, racial composition, socioeconomic 
status, and so forth. 

From manual or by combining information from various sources, what is the 
intended or appropriate grade level for use (elementary, middle, high school, 
college)? 

Provide information on any national, representative samples that can be used for 
comparison purposes. 


9. Description of constructs measured 
Quote from text with page citations 


Describe here what the engagement measure is described as measuring. Use 
quotes where possible, and cite page numbers. 

Obtain a copy of the instrument and attach. 

If the instrument has one or more subscales that are all related to engagement, give 
their titles. For each of these, cite sample items. 


10. Description of scoring/subscales/ 
reporting structure for engagement 
measure 


Describe here how the scoring/reporting occurs for the engagement measure. 
List engagement-related subscales with the number of items on each scale. 
Indicate the response format, for example, true/false, Likert-type, multiple choice. 


Mention unit of measurement 
(student, classroom, and so forth) 


open ended. For Likert-type, cite the rating scale (for example, 0 to 4, where 0 is 
strongly disagree and 4 is strongly agree). Indicate if there are reverse-scored items 
included. 

How are scores computed? 


11. Is the measure subject specific? 


□ No 

□ Yes, math 

□ Yes, science 

□ Yes, reading 

□ Yes, (other) 


12. Description of instrument method 

Describe relevant details of 
administration 


□ Student self-report 

□ Teacher rating of student 

□ Observation 


Describe variations in 
administration/guidance in manual 
available 


□ Other ( ) 

Provide a brief description of the details of administration (frequency, timing, and 
so forth). 

Provide a brief description of variations in administration reported and type of 
guidance provided in materials. 


13. Description of logistics (Training 
required? How to access? Support 
available? Publicly available?) 


Provide any relevant information that potential users would need to know (training, 
cost, support provided online/by developers, and so forth). 


14. Reliability information reported by 
developer? 


List any of the following included in the manual or in studies done by the 

developing author: 

□ Internal consistency/split-half reliability: Measures the correlation between the 
set of items on the subscale. Look for Cronbach's alpha. 

□ Test-retest reliability: Correlation between two (or more) administrations of the 
same item, scale, or instrument at different times. 

□ Inter-rater reliability: Correlation of scores between two or more observers or 
agreement between observers who rate the same item or behavior. 



(CONTINUED) 
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TABLE B7 (CONTINUED) 

Instrument-documentation protocol 



Information domain Directions 



1 Information domain 


Directions 


15. Reliability information reported by 
others? 

List studies/type of reliability 


List any of the above reliability information provided in studies done by users of the 
instrument. 


16. Validity information reported by 
developers? 


List any of the following included in the manual or in studies done by the 
developing author: 


Criterion-related validity — 
concurrent and predictive 

(Does engagement measure 
correlate with behaviors or predict 
achievement or other important 
outcome measures expected to 
be related to engagement [for 
example, attendance]?) 


Describe any correlations reported between engagement measures and criterion- 
related behaviors (attendance, achievement, and so forth). 


Construct validity 

(Do scores operate as expected 
if they were measuring what is 
intended?) 


Indicate if a factor analysis was conducted to establish subscales and if the factor 
analyses supported the subscales used. 

Describe any correlations reported that show how the engagement measure relates 
to other constructs in expected ways or to other measures of the same construct. 

Describe how results from various methods for assessing engagement correlate. 


17. Validity information reported by 
others? 


List any of the above validity information provided in studies done by users of the 
instrument. 


18. Overview of use 


Using articles and additional sources included in the instrument folder, describe 
how the instrument has been used (diagnostic, evaluations, monitoring, basic 
research, and so forth) and by whom (for example, by developer, by researchers, by 
school districts). 


Source: Authors. 
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APPENDIX C 
STUDENT SELF-REPORT 

SUBSCALE INFORMATION subscale names used and sample item wording 

(table Cl), and shows the subscales, categorized by 

This appendix provides additional information the three dimensions of engagement, used across 

on the student self-report measures, including the student self-reports (table C2). 



TABLE Cl 

Student self-report subscales and sample items 



Instrument 


Subscales/number of items 


Sample item 


4-H Study for Positive Youth 


Behavioral engagement (5) 


"How often do you come to class unprepared?" 


Development 


Emotional engagement (5) 


"1 feel happy to be part of school." 




Cognitive engagement (5) 


"1 want to learn as much as 1 can in school." 


Academic Engagement Scale: 
Consortium on Chicago School 
Research (CCSR/AES) 


Academic engagement (4 item) 


"1 work hard to do my best in this class." 


Attitudes Towards Mathematics 


Self-regulation (12) 


"Before a quiz or exam, 1 plan out how to study the 


(ATM) Survey 




material." 




Deep cognitive strategy use (9) 


"1 work several examples of the same type of 
problem when studying mathematics so 1 can 
understand the problems better" 




Shallow cognitive strategy use (5) 


"1 find reviewing previously solved problems to be a 
good way to study for a test." 




Persistence (9) 


"If 1 have trouble understanding a problem, 1 go over 
it again until 1 understand it" 


Engagement versus Disaffection 


Behavioral engagement (5) 


"When I'm in class, 1 listen very carefully." 


with Learning (EvsD) 


Behavioral disaffection (5) 


"When I'm in class, 1 just act like I'm working." 




Emotional engagement (5) 


"1 enjoy learning new things in class." 




Emotional disaffection (7) 


"When we work on something in class, 1 feel 
discouraged." 


High School Survey of Student 


Cognitive/intellectual/academic 


"Thinking about this school year, how often have 


Engagement (HSSSE) 


engagement (65) 


you done each of the following? 

(a) asked questions in class 

(b) contributed to class discussions 

(c) made a class presentation 

(d) prepared a draft of a paper or assignment before 
turning it in 

(e) received prompt feedback from teachers on 
assignments or other class work." 




Social/behavioral/participatory 


"Thinking about this school year, how often have 




engagement (17) 


you done each of the following? 

(a) had conversations or worked on a project with 
at least one student of a race or ethnicity different 
from your own 

(b) picked on or bullied another student." 




Emotional engagement (39) 


"How do you feel about the following statements 
related to your high school? 

(a) Overall, 1 feel good about being in this school. 

(b) 1 care about this school. 

(c) 1 feel safe in this school. 

(d) 1 have a voice in classroom and/or school 
decisions." 



(CONTINUED) 
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TABLE Cl (CONTINUED) 

Student self-report subscales and sample items 



Instrument 


Subscales/number of items 


Sample item 


Identification with School 


Belongingness (9) 


"School is one of my favorite places to be 


Questionnaire (ISQ) 


Valuing (7) 


"Most of the things we learn in class are useless." 


Motivated Strategy and Learning 


Self-regulation (9) 


"1 outline the chapters in my book to help me study." 


Use Questionnaire (MSLQ) 


Cognitive strategy use (13) 


"1 ask myself questions to make sure 1 know the 
material that 1 have been studying." 


Motivation and Engagement Scale 


Valuing school (4) 


"Learning at school is important." 


(MES) 


Persistence (4) 


"If 1 can't understand my schoolwork, 1 keep trying 
until 1 do." 




Planning (4) 


"Before 1 start a project, 1 plan out how I'm going to 
do it." 




Study management (4) 


"When 1 do homework, 1 usually do it where 1 can 
concentrate best." 




Disengagement (4) 


"I've given up being interested in school." 


Research Assessment Package for 


Ongoing engagement (5) 


"1 work hard on my schoolwork." 


Schools (RAPS) 


Reaction to challenge (6) 


"When something bad happens to me in school, 1 
say the teacher did not cover the things on the test." 


School Engagement Measure 


Behavioral engagement (5) 


"1 pay attention in class." 


(SEM)-MacArthur 


Emotional engagement (6) 


"1 am interested in the work at school." 




Cognitive engagement (8) 


"When 1 read a book, 1 ask myself questions to make 
sure 1 understand what it is about." 


School Engagement Scale/ 
Questionnaire (SEQ) 


School engagement scale (4 in 3 
subject areas) 


"How much time do you put into homework each 
week, including reading assignments?" 


School Success Profile (SSP) 


School engagement (3) 


"1 find school fun and exciting." 




Trouble avoidance (11) 


"1 turned in a homework assignment late or not at 
all." 


Student Engagement Instrument 
(SEI) 


Psychological engagement: 
teacher student relationships (9) 


"Adults at my school listen to the students." 




Psychological engagement: peer 
support for learning (6) 


"1 have some friends at school." 




Psychological engagement: family 
support for learning (4) 


"My family/guardian(s) are there for me when 1 need 
them." 




Cognitive engagement: control 
and relevance of schoolwork (9) 


"The tests in my classes do a good job of measuring 
what I'm able to do." 




Cognitive engagement: future 
aspirations and goals (5) 


"1 am hopeful about my future." 


Student School Engagement 


Emotional engagement (16) 


"1 feel excited by the work in school." 


Survey (SSES) 


Cognitive engagement (22) 


"How important do you think an education is?" 




Behavioral engagement (7) 


"1 get in trouble at school." 



Source: Authors' compilation. 
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TABLE C2 

Subscales used by student self-report instruments, by engagement dimension 



Engagement dimension Subscale 



Behavioral 


Behavioral engagement 3 
Academic engagement 
Social/behavioral/participatory engagement 
Persistence 3 '* 3 

School Engagement Questionnaire 
Trouble avoidance 
Behavioral disaffection 
Ongoing engagement 0 
Disengagement 0 


Emotional 


Emotional engagement 3 
Emotional disaffection 
Valuing 3 
Belonging 

School engagement 

Psychological engagement — teacher-student relationships 
Psychological engagement — family support for learning 
Psychological engagement — peer support for learning 
Reaction to challenge 


Cognitive 


Cognitive engagement 3 
Self-regulation 3 
Cognitive strategy use 
Deep cognitive strategy use 
Shallow cognitive strategy use 
Planning 

Cognitive/intellectual/academic 
Study management 
Control and relevance of schoolwork 
Future aspirations and goals 



a. Used by more than one instrument. 

b. Also, considered an aspect of cognitive engagement. 

c. Contains items reflecting both emotional and behavioral engagement. 
Source: Authors' analysis of instrument subscales from table Cl . 
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